Any experiment starts with a hypothesis and samples from the population are drawn to test the hypothesis in order to arrive at a conclusion about an observed phenomenon in the population. In other words, the idea is to use the sample to establish some facts about the population. But to complete this loop, one necessarily needs confidence interval and the P-value.
P- value may be defined as the probability of getting a difference that is at least equal to the zero value of null hypothesis if it were true. I am not going to bother you with the crux of how to calculate your P-value. On the other hand, confidence interval, determined by your estimate and standard error, gives you a range of values within which lies your actual population parameter. Conventionally, most studies use 95% confidence interval (95% CI).
It is widely accepted to interpret the P-value by comparing it with threshold values such as 0.05 which is the commonly used threshold in many studies. P-value of 0.004 is considered statistically significant and thus small enough to justify the jettisoning of the null hypothesis. In essence, the smaller your P-value, the stronger the evidence against the null hypothesis.
Before I delve into the reason why it is exceedingly beneficial to use P-value and confidence interval in interpreting your result, I’d like to point out few snags with using only P-value in your interpretation:
- We lose out on potential medical significance of differences observed in small studies that are usually ignored because the P-value is more than 0.05. Rather, we should always consider the range of values for the differences provided by the confidence interval.
- Statistically significant (P<0.05) findings are invariably ascribed to the effect of the exposure or treatment, while we forget that, by definition, one in every 20 comparisons in which the null hypothesis is true will give us P<0.05.
- A lot of people make the assumption that all P<0.05 findings are medically important, but there is a caveat, given a sufficiently large sample size, even an extremely small difference in estimate in the sample will be detected as significantly different from the zero value of null hypothesis.
Let’s consider a drug trial for a cholesterol-lowering drug A with the following data: N=30; mean difference between treated group and control = -40mg/dl; 95%CI for difference=-118.4 to 38.4; P-value = 0.32. Normally, this would only be reported as non-significant and then abandoned. However, the 95% CI demonstrates a range of increase in the cholesterol level (showcasing the adverse effect of the drug) as well as a massive decrease in the level of cholesterol.
Using the same drug but with a larger sample size (N=3000), the researchers got the following result: mean difference=-40mg/dl;95% CI= -47.8 to -32.2; P<0.001. With P-value less than 0.05, this is considered statistically significant. This smaller P-value suggests the existence of strong evidence against the null hypothesis of no effect. The 95% confidence interval shows a mean cholesterol decrease between 32.2 and 47.8mg/dl, thus making the drug a viable option for lowering blood cholesterol level.
When you consider the two scenarios, the mean difference between the two sample sizes are the same (-40mg/dl), however, because trial 1 was smaller, it provided no evidence to jettison the null hypothesis of no treatment effect. My point is, just because we had a large P-value in the first trial does not mean that the null hypothesis is true. This fact can perhaps be best illustrated by this phrase, “Absence of evidence is not evidence of absence”. In addition, a large sample size is better at detecting effect of treatment.
Furthermore, a cholesterol lowering drug B was used in another clinical trial: sample size(N)=5000; mean difference=-5mg/dl; 95%CI = -8.9 to -1.1; P-value = 0.012. With the P-value less than 0.05, this would normally be considered statistically significant. But the CI suggests that the reduction is at most 8.9mg/dl and as small as 1.1mg/dl. The size of the reduction is quite small for any clinical importance. Even though the P-value shows strong evidence against the null hypothesis, the CI shows it to be medically non-significant. Thus, it is imperative to evaluate the CI in order to ascertain the range of values for the mean difference.
In conclusion, in reporting your study, I’ll suggest you present the exact value of your P-value together with the 95% confidence interval, and the result of your analysis interpreted in the light of both concepts.