Variance Estimation of the Survey-Weighted Kappa Measure of Agreement
Feder, M. (2006, August). Variance Estimation of the Survey-Weighted Kappa Measure of Agreement. Presented at Joint Statistical Meetings, Seattle, WA.
The National Survey of Drug Use and Health (NSDUH) is currently conducting a study to assess the reliability of responses. An interview/re-interview method is employed where individuals are interviewed on two occasions. Reliability is assessed by comparing the responses from the two interviews. To measure the reliability of categorical responses, the kappa (?) index of inter-rater reliability is used. This measure is the statistic most often used to assess reliability of categorical variables. The common asymptotic variance estimation assumes simple random sample. The NSDUH sample design is complex, affecting the point estimates of ? and its variance. While correcting the point estimate of ? for the design is straightforward, the variance estimate is more involved. We present a Taylor linearization derivation, along with numerical results and comparisons with the asymptotic formula.