Evidence Reports

The high-quality literature search and the methodological rigor brought to the analysis of that literature are the essentials of sound evidence reports. To that must be added technically and clinically sophisticated interpretation of the findings, so that potential users will have a clear basis from which to work in formulating complete practice guidelines and recommendations.

Researchers and clinicians at both RTI and UNC have been in the forefront of work in the broad areas of quality of care, technology assessment, practice guidelines, health care organization and financing, clinical and biomedical investigation, and health care policy. They are well positioned to implement the stages of production and dissemination of evidence reports and like instruments.

The stages of production and dissemination can be divided as follows:

Topic Refinement

Topics selected for evidence reports need to be important and clearly defined. Because resources for producing evidence reports are limited for all public and private organizations, selecting topics in a systematic and easily defensible way is crucial.

Often, topics for AHRQ evidence reports are nominated by outside, private sector organizations. This process, mandated in a Federal Register announcement, is described on the AHRQ web site.

Literature Review

The high quality of the literature search and the methodological rigor brought to the analysis of that literature are essential to sound evidence reports and any practice guidelines that might emanate from them.

The RTI-UNC EPC brings significant library search and retrieval capabilities to this effort with access to the staff and computer resources of three major university libraries (UNC and, through RTI, those at Duke University and North Carolina State University). One special element of this is the pioneering work done at UNC's Cecil Sheps Center to develop HSRProj a computerized compilation of all ongoing (i.e., not yet published) work in the areas of health services, outcomes, and effectiveness research.

The scientific standards for this part of the development of evidence tables and reports have risen with time. Key elements now include clear analytic frameworks to point to significant patient care questions for the evidence report, well-crafted inclusion/exclusion criteria for the literature search (giving due attention to gray literature and unpublished materials), and standardized methods for grading the quality of the evidence.

In addition, issues of bias in the evidence and scientific criteria for meta-analyses are prominent design elements in conducting systematic searches and analyses. Understanding of the need for reliable, reproducible, yet efficient, processes for analyzing the literature has grown. This includes grading the quality of articles and studies in the literature as well as rating the strength of entire bodies of scientific evidence according to some accepted approaches, such as those of the U.S. Preventive Services Task Force or other groups. Also critical are developing efficient and, ideally, computer-based methods for such tasks as abstracting articles and maintaining bibliographies.

Supplemental Analysis

Major emphasis is placed on the methodological standards brought to the analysis of the literature. Consistent with this, high standards must be met in quantitative analyses of data and information from that literature. When appropriate, meta-analysis is a cornerstone for authoritative evidence syntheses reports. RTI-UNC EPC personnel are cognizant of advances in statistical and analytic techniques for meta-analyses, have contributed to these fields, and are highly skilled in applying meta-analytic techniques.

The team has experience in the use of a range of cost analysis methods that include cost of illness, cost minimization, cost effectiveness, and cost-benefit analyses; application of quality-adjusted life year approaches; use of complex decision analysis; and the like. A review of best approaches within the EPCs for applying methods for reviewing cost-effectiveness studies was published in June 2005 in the Annals of Internal Medicine. Moreover, our team has experience in developing customized software to support analysis, including software for meta-analysis, decision-tree, and quality-of-life studies.

Synthesis and Development of a Draft Evidence Report

The same level of rigor must be applied to summarizing the findings of the literature review and any supplemental analyses. A technically adept literature search and review is a necessary, but not sufficient, element of the production of authoritative evidence reports. To that must be added technically and clinically sophisticated interpretation of the findings, so that potential users will have a clear basis from which to work in formulating complete practice guidelines and recommendations. For example, it will be critical to make the linkages between findings and the quality of evidence abundantly clear.

The RTI-UNC Center staff has decades of experience with publishing peer-reviewed materials. We are familiar with the procedures, and intellectual rigor, required to develop coherent, well-organized, and clearly written evidence reports and other materials. Both partners also have skilled editorial and graphic design staffs who can ensure that narrative text communicates methods and results, and that graphical displays (which are of special importance for evidence reports) are unambiguous and appropriate to the findings being depicted.

Peer Review

Expert peer review serves several purposes. For example, it facilitates applying a multidisciplinary approach to the effort, especially in circumstances in which involving a full range of professional and lay/consumer perspectives in the analysis itself is not feasible. We give considerable attention to identifying an appropriate group of reviewers other than those from federal agencies whom AHRQ staff will contact. Most importantly, we give all written reviews thoughtful attention in revising draft report, and all such responses are documented.

Dissemination and Additional Reporting

The AHRQ retains final responsibility for the actual public release and dissemination to the public of EPC evidence reports and similar materials produced through their contracts. The EPC staff, however, provide assistance and ideas for dissemination, review news releases, create appropriate summaries of the longer evidence reports, give lectures, presentations, and workshops on the substance of our report, and otherwise contribute as much as possible to dissemination of this work.

The RTI-UNC Center has vast experience in publishing in the peer-reviewed health literature; our work appears in nationally and internationally recognized journals in health policy, health services research, medicine, nursing, dentistry, pharmacy, pharmacoeconomics, and public health. We are well prepared to draft and submit articles on both our findings and our methods to national general interest journals.