Area under the likelihood curve as a measure of evidential weight in nonsignificant meta-analyses

Abstract

OBJECTIVES: To quantify the distribution of evidential weight relative to the value of no difference or clinically meaningful thresholds in meta-analyses yielding P-values between .05 and .20.

STUDY DESIGN AND SETTING: We searched the Cochrane Library for systematic reviews with meta-analyses published between January 2025 and February 2026. For each meta-analysis, we calculated the area under the likelihood curve. In the primary analysis, we quantified the dominant proportion of evidential weight on either side of the value of no difference. In the secondary analysis, restricted to estimates with applicable minimal important difference thresholds, we partitioned evidential weight into three clinically interpretable regions: benefit (effects exceeding the minimal important difference in the clinically favorable direction), uncertain (effects between the minimal important difference and the value of no difference), and unfavorable (effects beyond the value of no difference in the direction opposite to clinical benefit).

RESULTS: We identified 1108 meta-analyses from 197 systematic reviews. Evidential weight favored one direction across all estimates, with a mean proportion of 94.0% on the dominant side. In the secondary analysis, restricted to 854 meta-analyses with an applicable threshold, the proportion of evidential weight in the uncertain region increased from 21.1%-56.1% at the small and large thresholds, while the benefit region decreased from 43.6%-8.7%. The unfavorable region showed a wide distribution with a mean of 35.2%.

CONCLUSION: Meta-analyses with P-values between .05 and .20 might show directional effects. There is a need to reconsider the binary classification of statistical significance in evidence synthesis.

PLAIN LANGUAGE SUMMARY: Researchers often treat study results as "negative" when the data simply do not provide strong enough evidence to rule out chance as the explanation (called statistically nonsignificant), but this can be misleading. We looked at 1108 meta-analyses from 197 Cochrane reviews with P-values between .05 and .20 to see what the data actually showed. We used a likelihood-based method that measures how much the evidence supports one direction of effect over another. In almost all cases, the evidence favored one direction. When we looked at clinical magnitude, the effects were typically small rather than large, but the direction of the evidence was clear and consistent. This means the findings gave useful clues about the direction of the effect, even though the effects were small. Overall, treating such results as "negative" just because they are not statistically significant can overlook important information. Adopting this likelihood-based approach could help researchers make better use of available evidence.