After long hours of data analysis and statistical tests, you finally determine that the outcome in the treatment group was statistically better than that of the control group. But before you can jump to any conclusions, you first have to determine whether the results are clinically significant. In other words, does it really matter? In this article, we’ll discuss why you should consider both statistical significance and clinical significance in your results analysis.
Statistical Significance
In clinical research, such as an investigator initiated trial, statistical significance helps us determine whether the results of a treatment group are different than that of a control group. To do this, we select the appropriate statistical tests based on our sample and then calculate the corresponding P-values. It is generally accepted that for P-values < 0.05, the differences between the groups are statistically significant. For more information on which tests you need for your analysis, please refer to our article on statistics. But what exactly is a P-value, and what does it really represent?
In order to understand P-values, we first have to review the concept of the null hypothesis. In every study, there is at least one key hypothesis that is being tested. For example, you hypothesize that a new eye drop formulation will significantly improve comfort for dry eye patients (as rated by a questionnaire) compared to another commercial product. In this scenario, there are three possible outcomes in this study:
- The new eye drop is more comfortable than the commercial eye drop
- The new eye drop is less comfortable than the commercial eye drop
- The new eye drop has the same level of comfort as the commercial eye drop (null hypothesis)
The third outcome in this scenario is considered the null hypothesis, in which there was no difference between the two tested groups.
Let’s hypothetically assume that the results of the study showed that were no differences in comfort scores between the two eye drops. However, due to random sample error, the sample did not represent the true population. If you were to repeat this study several more times, you would have found that there were differences between the two eye drops. To account for the potential errors in random sampling, we need to determine the P-values.
The P-values tell us how well the sample data reflects the null hypothesis. The higher the P-value, the more likely your data supports the null hypothesis. The lower the P-value, the more likely your data rejects the null hypothesis. For instance, a P-value less than 0.05 would suggest that there was less than a 5% chance that the null hypothesis was true, in which case you would reject the null hypothesis. In other words, for P < 0.05, the statistics would support that your initial hypothesis was true and that there was a difference in comfort between the two eye drops.
Clinical Significance
Remember that a P-value from statistical tests can only determine if there are differences between two groups. It does not tell you whether one treatment group was better or worse than another group, or if the differences are actually clinically relevant. Remember that just because something is statistically significant does not necessarily mean it’s clinically important.
Clinical significance measures the extent that a change can create a meaningful response for the patient. For example, you determined that a new eye drop formulation improved comfort in dry eye patients by 1% compared to another eye drop. Even if this result was statistically significant, a mere improvement by just 1% is not considered clinically significant. After all, would you buy or use the eye drop if it was only 1% better than a competitor’s product? Probably not!
Unlike statistical significance, there are no set guidelines for what is considered clinically important. The process of identifying whether results are clinically relevant, more often than not, will rely on your subjective judgement. In other words, your experiences and knowledge of the field are critical in helping you decide whether the results are clinically important. If you need more help deciding whether your results are clinically significant, one approach is to see refer to previously published clinical data and observations relevant to your study. Alternatively, you can consult the researchers involved in the clinical study or experts in your field for their opinions. If you need more help with interpreting your data, contact Sengi.
Interpreting data, whether that be from your clinical study or from a peer-reviewed source, is an art-form that gets better with experience. Many researchers will rush to conclusions based on statistically significant data without thinking about whether or not the results actually matter. Try to think critically and beyond the scope of the study – do the results actually matter, and can they be translated to clinical practice.
