Supplementing RDD Surveys with Web Data
Krotki, K., & Trofimovich, L. (2006, May). Supplementing RDD Surveys with Web Data. Presented at American Association for Public Opinion Research 61st Annual Conference, Montréal, Canada.
As telephone survey response rates continue to decline it becomes increasingly important to seek cost-effective alternatives to the telephone survey mode. One candidate is the web-based survey which, even though it is not based on a probability sample, comes from a frame that in many respects resembles the entire population. Furthermore, biases can be identified, measured, and to some extent controlled through response propensity weighting adjustment.
The core of the survey process remains the traditional and widely-accepted probability-based survey but with response rates declining, the question arises as to whether this factor is jeopardizing the very reason for implementing expensive survey methods such as RDD. When many of our response rates are below 60%, one begins to wonder what kinds of bias are creeping into our survey data and whether we are fully and correctly adjusting for these biases. It might be time to consider radical alternatives that would build on the probability-based methods but would supplement them with more novel methods.
The approach studied in this research is to supplement RDD surveys with data from a web-based survey, which is much less expensive than telephone surveys. This allows for significant increases in sample size at very low cost with the important caveat that we still have not developed methodologies for treating these combined mixed-sample survey data.
The specific goals of project are, first, to study the differences between the RDD and web-based survey data, using first unweighted and then weighted data. Second, we develop methodologies for combining the two sets of weights in a way that accurately reflects the underlying population and the processes that were used to develop the two samples. Finally, we develop design- and model-based strategies for the estimation process including variance calculations and significance testing.
This will be, to our knowledge, the first empirical, and statistically rigorous, comparison of non-probability web-based surveys, and RDD survey efforts. It will not only provide a head to head comparison of the two methodologies, but it will be one of the first studies to empirically investigate the ways in which these two modes of data collection can compliment one another.