As survey methods evolve, researchers require a comprehensive understanding of the error sources in their data. Comparative studies, which assess differences between the estimates from emerging survey methods and those from traditional surveys, are a popular tool for evaluating total error; however, they do not provide insight on the contributing error sources themselves. The Total Survey Error (TSE) framework is a natural fit for evaluations that examine survey error components across multiple data sources. In this article, we present a case study that demonstrates how the TSE framework can support both qualitative and quantitative evaluations comparing probability and nonprobability surveys. Our case study focuses on five internet panels that are intended to represent the US population and are used to measure health statistics. For these panels, we analyze the total survey error in two ways: (1) using a qualitative assessment that describes how panel construction and management methods may introduce error and (2) using a quantitative assessment that estimates and partitions the total error for two probability-based panels into coverage error and nonresponse error. This work can serve as a "proof of concept" for how the TSE framework may be applied to understand and compare the error structure of probability and nonprobability surveys. For those working specifically with internet panels, our findings will further provide an example of how researchers may choose the panel option best suited to their study aims and help vendors prioritize areas of improvement.
A process for decomposing total survey error in probability and nonprobability surveys
A case study comparing health statistics in US internet panels
Unangst, J. J., Amaya, A. E., Sanders, H. L. P. S., Howard-Doering, J., Ferrell, A. R., Karon, S. L., & Dever, J. A. (2020). A process for decomposing total survey error in probability and nonprobability surveys: A case study comparing health statistics in US internet panels. Journal of Survey Statistics and Methodology, 8(1), 62-88. https://doi.org/10.1093/jssam/smz040
To contact an RTI author, request a report, or for additional information about publications by our experts, send us your request.
Multifaceted risk for non-suicidal self-injury only versus suicide attempt in a population-based cohort of adults
Long-term effects of a diet supplement containing Cannabis sativa oil and Boswellia serrata in dogs with osteoarthritis following physiotherapy treatments
Pregnancy rates and clinical outcomes among women living with HIV enrolled in HPTN 052