Understanding Scientific Studies
Sometimes when reading a scientific study or a journal article about a scientific study, it feels as if the topic is so out of my realm of expertise that I can’t tell if it is a good experiment or not. While it is true that expertise in the field makes it much easier to interpret findings, there are a few basic facts I keep in mind when thinking about these topics to help determine if the experiment and results are reasonable, and to what circumstances they should apply.
What is the sample size?
The Law of Large Numbers says that the more times you repeat an experiment, the more likely the average result will be the expected value. In practice, this means the more samples you can get, the better. This makes sense logically - you would not be surprised if you flipped a coin twice and got heads both times, but you would be shocked if you flipped a coin 200 times and got heads every time. It is generally accepted in the scientific world that you need at least 40 samples per class to return statistically significant results; however, what constitutes a “good” number of samples will vary heavily based off the subject of study. A study that investigates left-handed people’s ability to use scissors that only has 40 participants is suspicious - there are a lot of left-handed people in the world. However, a study of the ability of people with ALS to use scissors that only has 40 participants is more reasonable, since thankfully only a small percentage of the population has ALS.
Has the study been peer-reviewed?
The intricacies of the scientific publishing industry are complex enough to warrant their own article, but there are some basic things you can keep in mind without additional context. The first thing to check is if the study is a pre-print or if it has been peer-reviewed. After submitting an article to a journal for publication, the author’s names and affiliations are stripped from the manuscript and it is sent to experts in the field for peer review. The reviewers will (rarely) choose to accept the manuscript as-is, suggest modifications before publications, or outright reject it. After all modifications have been made, the article is considered peer-reviewed and can be published in the journal. However, in some cases a pre-print can be published, in which the original manuscript can be published online before the review. There are some advantages to this, such as quickly disseminating findings during a pandemic, but pre-prints are by nature less reputable than peer-reviewed journals because no one has checked the work. There are many complex issues surrounding the peer review process, and sometimes bad research can slip by, but as a general rule something should not be taken as scientific fact without peer review, and something that has been peer reviewed is more reliable than something that has not.
Has the study been reproduced?
Similar to the Law of Large numbers and the peer-review questions, the more times an experiment had been repeated and the same results are found, the more likely it is that it is a sound finding. Many studies have found a link between smoking and cancer, and in general we take this knowledge as fact. Alternatively, the study “proving” that vaccines cause autism has not been replicated successfully and it was proven that the data was falsified; thus, we should not believe that finding. Of course, someone has to do an experiment and communicate the results for the first time in order for science to advance, but it is important to not immediately take that finding as gospel truth.
What kind of journal is this study being published in?
Unfortunately, in many cases, if scientists want their work to be open-access and free to the public, they need to pay the publishers. In addition, the number of published papers is often a factor in tenure decisions, so there are incentives for academics to publish as much as possible. This has led to a host of predatory journals that will publish papers of any quality just for profit. You can check if a journal is predatory by looking at websites that list reportedly predatory journals. In addition, you can check the “impact factor” of the journal. The impact factor is a measure of the frequency with which the average article in a journal has been cited in a particular year. A larger impact factor does not necessarily ensure that a journal is not predatory, but it is a good sign. Another factor to consider is if the subject of the journal aligns well with the subject of the article - a paper called “Investigating The Effect of School Lunches on Kid’s Self-Esteem” would make sense in the American Journal of Psychiatry, but if I see it in IEEE Robotics and Automation, I am suspicious that the editor is using their platform to push their own agenda.
Do the authors have any conflicts of interest?
In reputable journals, the authors are required to report any potential conflicts of interest. Conflicts could include employment or funding by a company that is affected by the study, marriage to someone who is employed by said company, or patents related to the field of research. These conflicts will be published with the paper, and occasionally it will be mentioned which part of the paper the person with the conflict worked on. Usually these conflicts are benign - when I published work related to my master’s thesis, one of my co-authors had to file a conflict of interest form because she was employed by the company that made the microscope we used in our experiments, but the reason she worked with us at all was because she was the expert on the microscope. Also, due to the way scientific funding is set up in America, often companies are required to do their own research before bringing products such as new drugs to market, so conflicts are common. This is not immediately bad, but it can become so if the company inflates results for their own purposes. For example, I would be more suspicious of a paper headline that reads “Company A’s drug cures cancer! (study funded by Company A)” than of one that reads “Company A’s new drug shows 12% decline in relapse of childhood leukemia after 5 years (study funded by Company A)”