Do Changes in Item Formatting and Question Placement Help Older Survey Respondents Provide Better Data? Improving Data Quality on the Medicare CAHPS Disenrollment Reasons Survey
Lynch, J. T., Kenyon, A. E., Wang, J., Rizk, S. C., Scheffler, S., Jackson, K. R., & Duke, M. C. (2005, May). Do Changes in Item Formatting and Question Placement Help Older Survey Respondents Provide Better Data? Improving Data Quality on the Medicare CAHPS Disenrollment Reasons Survey. Presented at American Association for Public Opinion Research 60th Annual Conference, Miami Beach, FL.
The Medicare CAHPS® Disenrollment Reasons Survey collects data from Medicare beneficiaries about the reasons they disenroll from their Medicare managed care health plan. In the survey, beneficiaries are first asked a series of 34 closed-ended items about reasons they disenrolled from their plan, after which they are asked to report their one “most important reason” for leaving their plan in an item that uses an open-ended response format. Responses from the open-ended “most important reason” item are coded to general categories related to quality of care and costs and benefits. The “most important reason” item is one of the most critical analytic items in the survey. The survey results are used to help explicate disenrollment rates, which the Centers for Medicare & Medicaid Services (CMS) is required to provide to Medicare beneficiaries. Survey results are also used by Medicare managed care organizations for use in their quality improvement initiatives and by CMS to monitor plan performance.
In this paper, we examine whether formatting and placement changes to the survey items using an open-ended response format improve the quality of the data collected in this survey of elderly respondents. We compare data collected in two survey years before and after these formatting and placement changes were made. We examine rates of missing data, interactions between the “most important reason” and the “other reasons for leaving” items, ability of older respondents to provide a single reason versus multiple reasons as their most important reason for leaving the plan, and the percentage of narrative responses that are uncodable. Our hypothesis is that making some targeted changes to open-ended survey items will improve the quality of the data collected from older respondents, leading to better information being available for beneficiaries making health plan choices.