Interview Timing Data: Simple yet Powerful Survey Instrument Development Tools
Heuer, R. E., Doherty, J. M., & Zweig, E. (2007, May). Interview Timing Data: Simple yet Powerful Survey Instrument Development Tools. Presented at American Association for Public Opinion Research Conference, Anaheim, CA.
Timing data in a computer-assisted telephone interview (CATI)/Web survey is not difficult to collect and is quite useful for subsequent questionnaire development. Time stamp information for predictable interview events (such as login/logout, form submission, interview completion) can be collected in several ways. Each provides different insights into the interview experience, and when used together, they present instrument designers with tools to fine-tune future instruments. Based on strategies used in numerous large-scale studies for the U.S. Department of Education, National Center for Education Statistics, this paper explores all aspects of implementing timers in a Web-based instrument as well as methodologies for analyzing and using those data to shape future questionnaire development. The different types of timers and strategies for determining when to implement each will be discussed. Methods of aggregating and analyzing timing information will be presented. When implemented and analyzed correctly, timing data contains substantially more information than simply the total interview completion times. For example, when designing future instruments that utilize the same or similar items, individual screen timers can be used to assist in predicting administration time and to provide good estimates of overall interview length. In addition, individual onscreen timing data coupled with data on help text usage or item-level missing data can be used to evaluate understandability of question or response-option wording and assist in determining response-option types (checkbox, radio button). Results from the National Postsecondary Student Aid Study (NPSAS), Beginning Postsecondary Students Longitudinal Study (BPS), Baccalaureate and Beyond Longitudinal Study (B&B), and National Study of Postsecondary Faculty (NSOPF) will be presented.