Critically Evaluating Medical Claims

When conducting online health research, you will encounter information about new medical studies and promising new medical interventions. It is important to always view these claims with a critical eye and try to understand the study’s relevance and clinical significance. The claims should never be taken at face value.

A review by HealthNewsReview.org of thousands of health news articles found that many failed to discuss the reliability of underlying evidence or accurately explain the medical findings, risks, and costs. To help readers assess the veracity of medical claims, the National Center for Complementary and Integrative Health (NCCIH) recommends that readers ask several questions:

These questions appear in an interactive lesson created by NCCIH. They are reproduced with permission in an abridged form below.

Was the study a controlled clinical trial?

In a controlled clinical trial, investigators compare the effects of different treatments in groups of study participants who are as similar as possible. For example, the outcomes in one group of participants who receive a new “experimental” treatment may be compared with the results of another group who received standard care, the “control group.” In effect the control group provides a comparison point for measuring the effects of the new treatment. In this case standard care is the “control” intervention.

There are many kinds of control groups. Ideally, participants are assigned randomly to one of the study groups. This helps ensure that the two groups are as identical in all respects as possible except for the intervention they receive. Other kinds of control groups are sometimes used, but they have an increased likelihood that factors other than the intervention affected the results.

How large was the study?

Studies with large numbers of people generally produce results that are more reliable than studies with small pools of participants. Larger studies can increase the accuracy of the study findings and reduce the probability that any effect observed in the study was due to chance. A study with too few participants may make yield inconclusive results. Statisticians and scientists have tools to determine how many volunteers are needed for a clinical study to be meaningful.

What did the control group receive?

In placebo-controlled trials, the control group receives an inactive treatment designed to resemble the treatment being studied. One example of a placebo is a pill that is medically inert (inactive) but looks like the experimental medicine being studied. Another example, called a sham, is used when the treatment being studied is a procedure (e.g., acupuncture), not a product. A sham procedure is designed to simulate the active treatment but does not have any active treatment qualities.

When possible, the “placebo” treatment and “experimental” active treatment are delivered in a “double-blind” fashion where neither the investigator delivering the treatment nor the volunteer know what they are getting. This reduces the possibility the volunteers know what they are receiving.

Have steps been taken to minimize bias?

It can be surprisingly difficult to avoid bias in clinical trials. For example, if patients know what treatment they received or if the investigators know which treatment a patient received, it may affect their impression of whether the patient improved–no matter how hard they try to avoid it. Therefore, it's important to ask what steps were taken to minimize bias. For example, was the trial “blinded” or “masked” so that neither the participants nor the investigators knew who was receiving which treatment? Researchers must always work to make sure that the study is objective and the results reflect the data accurately.

Are there potential conflicts of interest?

In viewing the results of any study, it is important to look for potential conflicts of interest or other sources of bias. It is useful to understand who funded the study. How removed were the sponsor and the investigators from any financial or reputational “stake” in the study outcome? Is there similar evidence from other independent sources? Fortunately, most medical journal articles now include information about relevant financial relationships.

How do the reported results compare with previous studies?

The strongest evidence about whether an intervention is useful and safe consists of results from several studies by different investigators. Rarely does a single study provide a final, definitive answer. There is a need for a study to be replicated, which involves repeating a study using the same methods but with different volunteers and investigators. Replication of a study gives more confidence that the results are reliable and valid. In addition, independent evaluations that compile the results of multiple studies and rigorously assess the quality of the data from them are especially useful. These evaluations are called systematic reviews and meta-analyses.

What is the meaning of statistical vs. clinical significance?

“Statistically significant” means the finding in the difference between the study groups is not likely to be due to chance. “Clinically significant” is a measure of the size of the effects observed in the study. For example, a study can find statistically significant differences between two treatment groups, but the differences are so small that they do not have clinical significance in terms of usefulness for patients or safety.

How old is the study?

Always look at the date of the study. Was it conducted in the last few years? Have there been more recent studies? You can search the National Library of Medicine’s PubMed for published studies. Sometimes, new research can dramatically change scientists’ view of a topic. For example, an early study may have suggested that a particular approach may be helpful for a certain medical problem, but a new, large clinical trial might show that it doesn’t have beneficial effects.

Questions originally published by the National Center for Complementary and Integrative Health (NCCIH). Questions to Help You Make Sense of Health Research. Retrieved on May 20, 2019. Content has been edited for brevity.