Approaches for reporting and interpreting statistically nonsignificant findings in evidence syntheses

Abstract

OBJECTIVE: To systematically review approaches for reporting and interpreting statistically nonsignificant findings with clinical relevance in evidence synthesis, and to assess their methodological quality and the extent of their empirical validation.

STUDY DESIGN AND SETTING: We searched Ovid MEDLINE ALL, Scopus, PsycInfo, Library of Guidance for Health Scientists, and MathSciNet for published studies in English from January 1, 2000, to January 30, 2025 for (1) best practices in guidance documents for evidence synthesis when interpreting clinically relevant nonsignificant findings, (2) statistical methods to support the interpretation, and (3) reporting practices. To identify relevant reporting guidelines, we also searched the Enhancing the QUAlity and Transparency Of health Research Network. The quality assessment applied the Mixed Methods Appraisal Tool, Appraisal tool for Cross-Sectional Studies, and checklists for expert opinion and systematic reviews from the Joanna Briggs Institute. At least two reviewers independently conducted all procedures, and a large language model facilitated data extraction and quality appraisal.

RESULTS: Of the 5332 records, 37 were eligible for inclusion. Of these, 15 were editorials or opinion pieces, 9 addressed methods, 8 were cross-sectional or mixed-methods studies, 4 were journal guidance documents, and 1 was a systematic review. Twenty-seven records met the quality criteria of the appraisal tool relevant to their study design or publication type, while 10 records, comprising 1 systematic review, 2 editorials or opinion pieces, and 7 cross-sectional studies, did not. Relevant methodological approaches to evidence synthesis included utilization of uncertainty intervals and their integration with various statistical measures (15/37, 41%), Bayes factors (6/37, 16%), likelihood ratios (3/37, 8%), effect conversion measures (2/37, 5%), equivalence testing (2/37, 5%), modified Fisher's test (1/37, 3%), and reverse fragility index (1/37, 3%). Reporting practices included problematic "null acceptance" language (14/37, 38%), with some records discouraging the inappropriate claim of no effect based on nonsignificant findings (9/37, 24%). None of the proposed methods were empirically tested with interest holders.

CONCLUSION: Although various approaches have been proposed to improve the presentation and interpretation of statistically nonsignificant findings, a widely accepted consensus has not emerged, as these approaches have yet to be systematically tested for their practicality and validity. This review provides a comprehensive review of available methodological approaches spanning both the frequentist and Bayesian statistical frameworks and identifies critical gaps in empirical validation of some approaches, namely the lack of thresholds to guide the interpretation of results. These findings highlight the need for systematic testing of proposed methods with interest holders and the development of evidence-based guidance to support appropriate interpretation of nonsignificant results in evidence synthesis.