Following up with nonrespondents via mode switch and shortened questionnaire in an economic survey

Abstract

To evaluate and adjust for nonresponse bias in household surveys, many social science studies conduct follow-up surveys with nonrespondents. By recruiting additional respondents, the goal of nonresponse follow-up (NRFU) surveys is to reduce nonresponse bias and make the respondent pool more representative of the characteristics of the sample as a whole. Often a change of data collection mode or a shorter questionnaire is implemented to increase response rates. However, whether these design features actually reduce nonresponse bias is usually unknown. What is also unknown is the effect of NRFU studies on measurement error, particularly when interviewer- and self-administered modes are used to administer sensitive questions susceptible to misreporting. Few studies have explicitly examined the joint impact of nonresponse and measurement error bias in NRFU surveys due to the lack of auxiliary validation data about the respondents and nonrespondents. We overcome this deficiency in an economic survey initially administered by telephone with mail nonresponse follow-up and administrative validation records available for the entire sample. This situation permits the estimation of both nonresponse and measurement error bias before and after the NRFU. We find that the NRFU survey succeeds in bringing in respondents who differ from the telephone respondents, but that these additional respondents are not always representative of the final nonrespondents. This results in reduced nonresponse bias for some items, but increased nonresponse bias for others. We also find that combining the mail NRFU respondents with the telephone respondents reduces measurement error bias for economic estimates. Lastly, we report a paradoxical finding in which adding NRFU respondents to the respondent pool produces greater total bias in some survey estimates despite reducing both nonresponse and measurement error bias separately. We conclude with a discussion of the practical implications of these findings and speculate on their possible causes.