The Use of Vignettes in Evaluating Multilingual Questionnaires
Sha, M., & Pan, Y. (2009, May). The Use of Vignettes in Evaluating Multilingual Questionnaires. Presented at AAPOR 2009, .
Vignettes are hypothetical situations that can be used to review questionnaire items, whereas it would be cost prohibitive to recruit respondents that can represent all the situations in the questionnaire. The vignette method has been shown effective in evaluating questionnaires in English and Spanish (Caspar and Goerman, 2007), but little research is available on its application to other languages. This paper attempts to fill in the knowledge gap by examining the use of vignettes to evaluate questionnaire translation in Chinese, Korean, Vietnamese, and Russian.
This study is based on a cognitive testing project undertaken at the U.S. Census Bureau to pretest the translation of the 2010 Census form in multiple languages. A total of 93 speakers of Chinese, Korean, Vietnamese, and Russian languages were interviewed. As a comparison, 16 English interviews were conducted. We constructed vignettes to test the age reporting instructions and the relationship terms. Respondents were directed to consider each vignette and respond to the questionnaire item and followup probes about how they reached their answers.
This study provides a unique opportunity to examine the use of vignettes in evaluating questionnaire across multiple languages. We found that the vignettes led respondents to review a term when it did not apply to their situation and that the vignettes were useful in eliciting cultural implications of a term. For example, in the Asian languages, the term “unmarried partner” may carry an unintended connotation whereas it describes a common-law marriage in Russian. The vignette task itself, however, has its limitations. We learned that the descriptions needed to be short and simple and that the vignettes needed to conform to respondents’ cultural and linguistic expectations. This paper will examine findings across the four non-English languages and will explore future research directions.