Evaluating search approaches for a network meta-analysis

Abstract

BACKGROUND: MEDLINE, Embase, the Cochrane Central Register of Controlled Trials (CENTRAL), and reference lists are commonly used when searching randomized controlled trials (RCTs) for inclusion in a systematic review. Recently, a new database for RCTs, Epistemonikos Database of Trials (EDT), was launched in a pilot stage. This case study aimed to compare EDT, MEDLINE, Embase, CENTRAL, reference list checking, and database combinations for the identification of RCTs about pharmacological migraine treatments and to assess the impact of missed trials on network meta-analyses (NMA).

METHODS: We assessed the following: (1) the performance (recall, precision, number needed to read [NNR], F1- and F2-scores) of database searching, reference list checking, and database combinations compared to a relative recall reference standard, (2) the impact of differing search results on the findings of NMA (overall network estimates, certainty of evidence ratings) based on RCTs with low/moderate risk of bias, and (3) the suitability of EDT for systematic searching.

RESULTS: Recall was high for all assessed search approaches (98.3-99.4% for single-database searches without date limit, 99.4-100% for database combinations without date limit, and 93.3% for reference list checking of older systematic reviews combined with publication-date-limited database searching). However, NNR and F-Scores varied widely, ranging from an NNR of six and F2 of 0.5 for EDT alone to an NNR of 16 and F2 of 0.2 for the combination of reference list checking and database searching. Different search results led to similar network estimates and identical certainty-of-evidence assessments as the reference standard. While the search, export, and documentation functions of EDT were less sophisticated than other assessed databases/platforms, they were sufficient to conduct reproducible and comprehensive systematic searches.

CONCLUSIONS: All search approaches we tested were acceptable for an NMA of pharmacological migraine treatments; high search recall led to similar NMA results. However, the associated number of references to be screened at title/abstract level differed greatly. Specialized RCT databases (CENTRAL, EDT) were more efficient in this regard than general healthcare research databases (MEDLINE, Embase). Additionally, the pilot version of EDT proved to be a promising cost-free option for RCT searching.