Potential survey error due to a panel design

Sources of Non-sampling Error in the NCVS

The NCVS is a rotating panel survey conducted by the US Census Bureau that interviews households seven times at 6-month intervals (US Census Bureau, 2014). A rotating panel design offers three distinct advantages over a cross-sectional survey: (1) all interviews after the first are bounded by the previous interview, (2) cost savings are realized through higher response rates and the ability to switch to lower cost modes in later waves, and (3) it offers a longitudinal data structure for assessing change over time (Berzofsky & Carrillo-Garcia, 2019). However, each of these advantages is countered by the potential for non-sampling error because of the panel design. The three major sources of error are unbounded interviews and telescoping, mode effects, and fatigue.

Although this paper focuses on the three sources of error most associated with longitudinal or panel surveys, they are not the only sources of error under the TSE framework. Within the broader TSE framework, the types of longitudinal or panel error being examined fall under measurement error (e.g., telescoping and mode effects) and nonresponse error (e.g., fatigue). However, other sources of error—such as coverage error, sampling error, adjustment error, and processing error—are not discussed heavily here because they would apply to the NCVS regardless of the design type selected. In other words, the use of a panel design does not influence the impact that these other error sources will have on survey estimates. Therefore, their influence on the final estimate will be constant across designs. For this reason, we do not focus on them in this paper.

Mode Effects

The second potential source of non-sampling error in a panel survey is mode effects. Often, in panel surveys like the NCVS, a more expensive mode—an in-person interview—is used for the initial interview to help gain the cooperation of the household whereas a less expensive mode, like telephone interviewing, is used in subsequent interviews. Studies on mixed-mode designs have had conflicting results. Holbrook, Green, and Krosnick (2003) found that in-person interviews yield higher estimates than telephone interviews, whereas Cernat (2015) found that in-person and telephone interviews do not yield different results in a panel survey. NCVS interviewers strive to conduct in-person interviews for the first interview with a household or person and telephone interviews for follow-up waves whenever possible. About 44 percent of NCVS interviews are conducted by telephone.

Figure 1 presents violent victimization rates from 2007 to 2013 by mode of data collection for interview waves 2–7 in the NCVS. (Note that the findings in this figure look similar when all interview waves are included.) In-person interviews consistently yield higher victimization rates than telephone interviews, which indicates a possible mode effect. However, the results in Figure 1 do not control for factors like demographic characteristics or the length of time the person has been in the panel.

Figure 1

Violent victimization rates by mode of data collection and year, 2007–2013

Fatigue

The final potential source of non-sampling error we examine is fatigue. For this paper, fatigue is used as an umbrella term for multiple types of error that occur because of the repeated (or longitudinal) nature of the survey design. Types of error that fall under this umbrella include respondent fatigue and interviewer non-compliance. All sources of fatigue have the same impact on the resulting estimates—an under-reporting of the outcome of interest.

Respondent fatigue (also known as panel conditioning) can occur if respondents deliberately choose not to report incidents during subsequent interviews to reduce the interview duration (Hart, Rennison, & Gibson, 2005).

In the NCVS, for example, screener questions are used to identify whether the respondent has experienced a crime. An affirmative response to a screener question then triggers a detailed incident report, which is used to understand the nature of the incident and classify the type of crime. Once a respondent learns how much longer the interview becomes when an affirmative response is given to a screener question, they may choose not to report future incidents in subsequent interviews. However, respondent fatigue can be difficult to identify and quantify because even the act of taking the survey could cause a respondent to change their behavior and potentially avoid future victimizations (Cantor, 2007).

Interviewer non-compliance occurs when an interviewer rushes, does not administer the survey as designed, or uses information learned in past interviews to assume answers in the current wave. Rather than going through the entire survey instrument as intended, the interviewer leads the respondent to an answer or does not provide the opportunity to answer questions.

Figure 2 presents the violent victimization rates by interview number—the number of times a respondent participated in the survey regardless of the household’s time-in-sample (TIS) (i.e., the number of times a selected address has been in the sample). Victimization rates decrease as a respondent’s interview number increases, which may indicate fatigue.

Figure 2

Total violent crime rate by interview number

Theoretical Model for Assessing TSE

The NCVS introduces a new sample of about 50,000 households to the panel every six months to replace the households rotating out after being in sample for seven interview waves. Because of non-response, replacement households moving into an address, and persons aging into the sample, the number of interviews in which a respondent has participated may differ from the TIS.

Figure 3 presents the theoretical constant rate of being a victim of violent crime along with the NCVS observed rate, across TIS. Because participation in the NCVS is not correlated with victimization risk, the probability of being a victim of crime should not change based on a household’s TIS or respondent interview number. In other words, without any non-sampling error the expected or theoretical victimization rate at each interview should be relatively constant. However, Figure 4 shows that the observed victimization rate is higher at TIS-1 and lower across TIS-2 through TIS-7 than the theoretical rate.

The differences between the observed and theoretical model are caused by non-sampling error from unbounded interviews, mode effects, and fatigue. The first interviews for respondents represent a completely unbounded estimate that has no fatigue and minimal mode effects because most interviews are conducted in person—this would be the estimate produced by a cross-sectional survey. The second interview for a respondent represents a bounded estimate with potential mode effects and, assuming respondents become more fatigued with each subsequent interview, the least amount of potential fatigue. Because the likelihood of fatigue increases with each interview, it is hypothesized that the victimization rates at the first and second interviews represent the upper and lower bounds of the true victimization rate.

Figure 3

Observed and theoretical violent victimization rates by time-in-sample

Figure 4

Observed violent victimization rates and theoretical range for expected victimization rate by interview number

Approach

A two-step approach was used to examine the three major sources of potential non-sampling error in a longitudinal survey:

1. Assess each source of error independently and determine the best correction method, and

2. Develop a combined adjustment method based on the results of step 1.

As a working adjustment factor, it was anticipated that the combined TSE adjustment (${ADJ}_{TSE}$) would take the form

where ${ADJ}_{TELE}$is the adjustment for telescoping, ${ADJ}_{FAT}$ is the adjustment for fatigue, and ${ADJ}_{MODE}$ is the adjustment for the mode effect.

Data

This study used NCVS public use data files from 2007 to 2013 (see, for example, US Department of Justice, Bureau of Justice Statistics, 2006, 2013). The survey years 2007–2013 were used to develop and assess the alternative adjustments for each source of non-sampling error. In 2006, the NCVS sample went through a decennial redesign, and at the same time, BJS began using TIS1 data with a bounding adjustment in the survey estimates. Thus, 2007 was the first full year, post-redesign, where a bounding adjustment was applied to the NCVS. These years were also included because persons and households could be linked across survey years for the full period.¹1 NCVS public use files before 2007 cannot be linked at the household or person level for two reasons: (1) the scrambled ID number for a household changed during the phase-in/phase-out period in 2005 and 2006 when the 2000 Census primary sampling units are rotated in, and (2) before 2006, the scrambled ID numbers were hand entered by interviewers; ID numbers cannot fully be linked because of errors in the keying process.

The use of public use data introduces a couple limitations to our analysis. First, geographic identifiers are greatly limited with only indicator for Census region present on the file. Therefore, any attempt to examine if certain errors are concentrated to particular geographies or interviewers is not possible. Second, the public use file only contains the final adjusted analytic weight. Therefore, any adjustments we propose cannot begin with the design-based weight or some earlier weight before all the final adjustments are applied.

For the development of the fatigue adjustment, it was necessary to use data from addresses that were in sample for all seven waves. Thus, only sampled addresses that were rotated into the sample in 2007 or later and were rotated out of the sample by the end of 2013 (i.e., the selected address was in the sample for all 7 TISs) were included in the fatigue adjustment analyses; the sample and rotation groups included were Sample 24 rotation groups 5 and 6 and Sample 25. However, for the purpose of producing annual victimization rates, the analyses included all interviews conducted during that survey year as done by BJS when producing estimates.

Assessing Telescoping Error

When TIS-1 cases were included in the annual NCVS crime estimates in 2006, BJS and the Census Bureau recognized that this would increase the annual estimates because of telescoping and higher TIS-1 rates. To account for this, a bounding adjustment was applied to the weighted victimization rate in TIS-1, adjusting it to the average weighted victimization rate reported during TIS-2 through TIS-7. The bounding factor was based on the previous 12 months of data and calculated separately for each month.

The original bounding factor assumed that telescoping does not differ across demographic groups. However, in Figure 5, age is used to illustrate that the rate of telescoping is not the same across different subpopulations. In addition, the original adjustment was only applied to TIS-1 interviews but not to interviews unbounded for the six other reasons presented in Table 1. Thus, the goal of the assessment of telescoping error was to determine (1) if telescoping varies by demographic group and (2) how the inclusion of all sources of unbounded interviews impacts the adjustment factor and resulting estimates.

Figure 5

Bounded and unbounded violent crime rates by time-in-sample and age category

Assessing the Impact of Fatigue

Fatigue, which can cause respondents to suppress or reduce the reporting of victimizations, is assumed to be driven by exposure to the survey or a respondent’s interview number.

It is difficult to know with certainty if a respondent who previously reported a victimization reported fewer or no victimizations in a subsequent wave because they knew the survey would take longer with an affirmative response or because they were not victimized. Couzens, Berzofsky, and Krebs (2014) tried to distinguish between these two situations and was unable to identify definitively when a respondent who previously reported victimizations reported fewer incident in subsequent interviews because of fatigue.

Rather than trying to isolate fatigued respondents, another option is to apply an adjustment to all respondents who reported a victimization and who could potentially be fatigued. Based on the theoretical framework, the total number of victimizations reported by random samples of the population should be roughly consistent over the same time period, regardless of the sample’s TIS. Thus, a fatigue adjustment would inflate victimization weights in waves 2 through 7 to be consistent with model-estimated benchmark victimization levels that factor out the effect of fatigue because of survey exposure. Creating this inflationary adjustment factor is based on modeling the shape of the fatigue curve across interviews 2–7 and extrapolating that to interview 1, at least among bounded cases. This provides a fatigue-adjusted interview 1 rate that takes respondent characteristics into account and that is not affected by telescoping. Using the ratio of this estimated interview 1 rate over the interview number–specific rate estimated using the same model, we are able to reverse the deflationary effects of fatigue while accounting for the ways in which respondent characteristics affect the shape of the fatigue curve.

To adjust each respondent’s number of victimizations to account for fatigue, a model-based approach similar to the model-based telescoping adjustment was employed. Under this approach, a separate model was estimated for person crimes and household crimes. The dependent variable was the number of reported victimizations (including zero). Therefore, for each victimization type a Poisson model was used to regress the number of victimizations reported to the interview number (as a continuous variable), the square of the interview number, person or household characteristics associated with fatigue, and the interaction between those characteristics and interview number and the square of the interview number.

This method generated a separate fatigue adjustment factor for each combination of the independent variables. An individual’s fatigue factor is the rate ratio of the estimated unfatigued victimization count over the estimated fatigue-affected count. For example, if the model contained interview number, the square of interview number, and gender (female reference level), then our model would be as follows:

For a female respondent at wave number 4 the fatigue adjustment factor (FF) is

For a male respondent at wave number 7 the FF is

Because the model-based adjustment is based on multiple characteristics, some of the adjustment factors may be considered extreme. We assessed the adjustment factors for outliers and determined if Winsorization was necessary.

Assessing the Impact of the Mode Effect

To assess the impact of mode effect and to determine an appropriate adjustment, two modeling approaches were implemented: (1) Poisson model of the number of victimizations by mode type and (2) propensity score balancing (both matching and reweighting).

Poisson model. A Poisson model was developed to estimate the effect of mode on the number of reported victimizations, controlling for demographic characteristics and interview number.

Propensity score matching. Because mode is tied to non-response and survey exposure, and is, therefore nonrandom, propensity score models were used to balance the analysis sample on observed characteristics across mode groups. Propensity score matching and inverse treatment probability weighting were used to balance the analysis sample on person- and household-level characteristics and survey weights and to estimate adjusted and unadjusted victimization (Poisson) models within each interview number.

Developing a Combined TSE Adjustment

Once potential adjustments were developed for each of the three sources of error, the next step was to implement a process for assessing which of the resulting adjustments best aligned with the TSE theoretical model. A three-step evaluation was developed in which adjustment options that did not meet minimum standards were removed from consideration. The three-step process included the following:

Evaluation step 1: Determine if, using the total adjustment factor (product of telescoping adjustment factor and FF), victimization rates are consistent across interview wave by type of crime.

Evaluation step 2: Determine if victimization rates based on the total adjustment factor are consistent across interview number for detailed demographic characteristics.

Evaluation step 3: Determine if annual victimization rates across time, by type of crime, differ by adjustment factor.

Because some of the adjustment factors require up to 36 months of retrospective data, the evaluation of the combined adjustments used data from 2008 to create the adjustment factors and data from 2011 to 2013 to produce victimization rates.

Telescoping Error

Figure 6 presents the results of the random forest analysis that was used to narrow down the characteristics used in the class-based and model-based adjustment factors. For both violent and property crime, we examined the results to see if there was a set of characteristics that clearly induced larger mean square error (MSE) increases than the other characteristics. For violent crime, a 35 percent MSE breaking point was chosen based on a subjective review of how the MSEs for each characteristic were clustered. The 35 percent MSE breaking point identified race/ethnicity, education, urbanicity, household income, and age for incorporation into the class-based and model-based adjustments. For property crime, an 80 percent MSE breaking point was chosen based on a subjective review of how the MSEs for each characteristic were clustered. This 80 percent MSE breaking point identified age urbanicity, household income, race/ethnicity, household tenure, and gender for incorporation into the class-based and mode-based adjustments.

For each alternative adjustment, a factor was produced based on all violent crimes and all property crimes. Considering the five bounding reference types and the results of the random forest analysis, 70 different alternative adjustments were produced. Each of these adjustment factors were reviewed for extreme values. No extreme values were identified for any of the telescoping adjustments.

Figure 6

Percent MSE for person and household characteristics based on random forest analysis for violent and property crimes

Fatigue Error

For the fatigue adjustment, the first step was to determine which characteristics to include in the model. The same characteristics identified during the random forest analysis (Figure 6) were used in the fatigue model.

Figure 7 presents the mean number of incidents reported by interview number before and after applying the fatigue adjustment. As expected, the number of incidents in interview numbers 2–7 was greater after the adjustment, but lower than the number incidents in interview number 1, which is subject to telescoping. The adjustment was successful in correcting the number of incidents to be similar in interview numbers 2–7.

Figure 7

Mean number of incidents by interview number, adjusted and unadjusted by fatigue adjustment

The fatigue adjustment factors for each respondent were reviewed for extreme values. Because extreme values were identified and could result in respondents from having too much influence on the estimates, a 5 percent Winsorization was applied. As exemplified in Figure 8, the Winsorization had a small impact on the overall victimization rate but had varying impact by domain category. For instance, estimates for persons 12–17 were largely impacted whereas those 65 and older were negligibly impacted. Notably, when more data are used to estimate the fatigue models (2011 had the fewest data points and 2013 the most), the impact of Winsorization decreases substantially. This illustrates the need for as many data as possible when fitting fatigue adjustment models.

Figure 8

Impact of 5% Winsorization in the fatigue-adjusted victimization rates on the violent crime rate for all persons, 12–17-year-olds, and persons 65 or older by year, 2011–2013

Mode Effect Error

Figure 9 presents the violent victimization rates by mode after controlling for interview number. After taking interview number into account, the apparent mode effect disappears (see also Couzens et al., 2014). Moreover, as shown in Figure 10, the victimization rates at each interview number are similar by mode, but the percentage of interviews conducted in each mode differs by interview number. Therefore, the apparent mode effect is largely a function of telescoping and fatigue and does not need to be accounted for separately in the combined TSE adjustment.

Figure 9

Violent victimization rate by data collection mode after controlling for interview number and demographic characteristics

Figure 10

Percentage of interviews conducted and victimization rate by mode and interview number

Combined TSE Adjustment

Given the findings of the independent reviews of the three major sources of error in the NCVS, assessment of the combined TSE adjustment consisted of 70 possible adjustments. Because no adjustment for mode was necessary, the combined TSE adjustment was modified to

Evaluation Steps 1 and 2

Evaluation step 1 assessed victimization rates by type of crime across TIS and interview number. A successful adjustment would produce a near straight line across TIS and interview number. However, as shown in Figure 11, the initial results led to victimization rates that were lower in TIS-1 than the remaining TISs. After an investigation, it was determined that applying the telescoping adjustments to victimization counts that had not been adjusted for fatigue potentially over-corrected for telescoping. Failing to account for fatigue in TISs 2–7 caused the apparent telescoping error to be artificially inflated. Therefore, the adjustment factor over-corrected for telescoping in TIS-1. Based on this finding, we modified the combined TSE adjustment to be

${ADJ}_{TSE}={ADJ}_{TELE|FAT}\times {ADJ}_{FAT}$

As shown in Figure 11, once this adjustment was made the victimization rates across TIS became more level. The adjustment for fatigue is implicitly conditioned on telescoping because it was based on bounded cases only.

Figure 11

Property crime victimization rates based on combined TSE adjusted before (dashed line) and after conditioning the telescoping adjustment on fatigue-adjusted incident counts by TIS

After adjusting the formula for the combined TSE adjustment, the overall victimization rates for violent and property crimes were compared for each of the five bounding types. As shown in Figure 12, the adjusted rates for both violent and property crimes were similar for bounding types 1–3 (the types based on TIS) and for bounding types 4 and 5 (the types based on whether an interview is bounded). This suggested that the focus of the next evaluation steps could be limited to bounding type 1 (TIS-1 vs. TIS-2 through TIS-7) and type 4 (unbounded vs. bounded from all TISs). These two types were chosen over the other bounding types because they incorporated all available data, which increases the power of the adjustment factors.

Figure 12

Violent and property victimization rates by interview number and bounding type

In Figures 13 and 14, the model-based telescoping adjustment and bounding types 1 and 4 are used to compute NCVS victimization rates for the major crime types by interview number. The figures show the unadjusted estimates (light gray lines) and the adjusted estimates (black lines). Across both bounding types and for all crime types, the adjustment resulted in trend lines that were closer to the theoretical model than the unadjusted estimates were. However, even with the adjustment the lines are not completely flat. To determine if the difference in the slopes between the adjusted trend line and the theoretical trend line, an adjusted F test was computed. The adjusted F tests for linear trend generally found the victimization rate trends to be flat (i.e., slope = 0), but that was sometimes a result of wide confidence intervals (CIs) at later interview numbers. In general, the adjustment factors performed better for property crimes than violent crimes. Because the adjusted trend lines appeared similar for bounding types 1 and 4 both were included in Evaluation Step 2, but the other three bounding types were no longer considered.

Figure 13

Adjusted (black line) and unadjusted (gray line) victimization rates by type of violent crime and interview number by bounding type 1 and 4 based on the model-based telescoping adjustment

Figure 14

Adjusted (black line) and unadjusted (gray line) victimization rates by type of property crime and interview number by bounding type 1 and 4 based on the model-based telescoping adjustment

The combined TSE adjustments based on the population-based, class-based, and model-based adjustments also performed similarly. Therefore, all three telescoping adjustment types were considered in Evaluation Step 2, which involved producing victimization rates by person and household characteristics of interest, interview number, and bounding type. The Evaluation Step 2 analysis showed that the adjustment performed well, except in cases where the sample size for a category was small. This held true across characteristics of interest that were and were not included in the models for violent and property crime (see the Appendix for Figures A.1, A.2, A.3, and A.4, which show rates of victimization across demographic and incident characteristics).

As with Evaluation Step 1, the combined TSE adjustments based on the population- and class-based telescoping adjustments (not shown) performed similarly to the TSE adjustments that used the model-based telescoping adjustment. Therefore, all three options were considered in Evaluation Step 3.

Evaluation Step 3

Figures 15, 16, 17, and 18 provide examples of comparisons among published victimization rates for 2011, 2012, and 2013 and victimization rates based on several alternative combined TSE adjustments. Across violent crime, serious violent crime, general property crime, and motor vehicle theft, the BJS published rate (i.e., the estimate based on the current bounding adjustment) was the lowest and the unadjusted rate (i.e., the estimate generated without any bounding or fatigue adjustment) was the second lowest rate for all survey years. The victimization rates based on the combined TSE alternative adjustments were about 1.5 to 2 times larger than the published victimization rate for each type of crime. However, the order of the alternative adjustments (from highest victimization rate to lowest) was not always the same. For example, for violent (Figure 15) and serious violent crime (Figure 16), the model-based telescoping adjustment produced one of the highest victimization rates, but for property crime (Figure 17) and motor vehicle theft (Figure 18) several of the class-based adjustments produced higher rates. Although the magnitude of the adjusted victimization rates varied across bounding reference types 1 and 4, the rankings of which adjustment produced the highest rates were largely consistent by crime type.

Figure 15

Violent crime victimization rates by select combined TSE adjustment factors, bounding reference type, and survey year, 2011–2013

Figure 16

Serious violent crime victimization rates by select combined TSE adjustment factors, bounding reference type, and survey year, 2011–2013

Figure 17

Property crime victimization rates by select combined TSE adjustment factors, bounding reference type, and survey year, 2011–2013

Figure 18

Motor vehicle theft crime victimization rates by select combined TSE adjustment factors, bounding reference type, and survey year, 2011–2013

For the most part, the trend line of the alternative adjustments (i.e., the slope of the victimization rates over time) followed the trend of the published estimate. This implies that the alternative adjustments only impacted the magnitude of the estimate.

Review of Error Because of the NCVS Design

At the onset of the analysis it was hypothesized that longitudinal survey designs lead to three potential sources of non-sampling error: (1) telescoping error, (2) fatigue, and (3) mode effect error.

Data from the NCVS show that unbounded interviews result in significantly higher victimization rates than bounded interviews. Based on this finding, we feel certain that respondents in unbounded interviews engage in telescoping. Therefore, an adjustment for all unbounded interviews is appropriate.

Although it was not possible to determine which respondents or interviewers were becoming fatigued (Couzens & Berzofsky, forthcoming), applying the theoretical model to NCVS data suggested the presence of fatigue in interviews 2 through 7. Not adjusting for fatigue while adjusting for telescoping only applies a downward adjustment without any compensating upward adjustment. From a TSE perspective, fatigue should be adjusted for based on the same rationale as telescoping.

Although an initial descriptive review of victimization rates by data collection mode indicated a mode effect (Figure 1), a more thorough assessment resulted in findings more in line with Cernat (2015) than Holbrook et al. (2003). In other words, the apparent mode effect was really a function of the mixed-mode protocol and the predominate use of in-person interviews in circumstances where the estimates were unbounded rather than differences in the victimization rates. Therefore, if telescoping and fatigue are already addressed, a further adjustment for mode is not necessary.

Determining the Best Adjustment

This evaluation shows the current NCVS adjustment results in underestimated victimization rates. However, it is less clear how to adjust the victimization rates appropriately. The results of Evaluation Step 3 suggested that the alternative combined TSE adjustment factors lead to a wide range of adjusted estimates. In determining the most appropriate factor for the NCVS and other longitudinal surveys, it is necessary to consider (1) the bounding reference period, (2) the telescoping adjustment method, and (3) the legitimacy of a fatigue adjustment.

In terms of the bounding reference period, Evaluation Step 1 found that the reference period comparing TIS-1 with the other TISs (type 1) and the bounding reference period that compares any unbounded interview with bounded interviews regardless of TIS (type 4) behaved similarly to the adjustments based only on data from TIS 1–4 (Types 2 and 5) or TIS 1–2 (Type 3). For this reason, adjustment types 1 and 4 were considered for further analysis as they were based on the most data and allowed for more refined adjustments in further steps, particularly for the class- and model-based approaches. Type 1 is consistent with the approach used in the current NCVS bounding adjustment whereas type 4 is more theoretical in determining that all unbounded interviews should be treated the same. As seen in Evaluation Step 3, the annual victimization rates based on the type 4 bounding reference period are generally lower than the estimates based on type 1. This is because more interviews have the downward telescoping adjustment applied. From a TSE perspective, if all bounded cases in TIS-2 through TIS-7 are upwardly adjusted for fatigue it follows that any unbounded cases should be adjusted as well. Therefore, the recommendation is to use bounding reference period type 4.

In terms of the telescoping adjustment method, it is less clear which is most appropriate. Because alternative combined TSE adjustments lead to a wide range of victimization rates and there is no known truth for the victimization rate—because it includes both reported and unreported victimizations—among the adjustments that meet our evaluation criteria in Evaluation Steps 1 and 2, the selection of the best telescoping adjustment is a bit subjective.

To assess the legitimacy of the fatigue adjustment, the analysis relied on the theoretical model rather than attempting to identify which respondents experienced fatigue. Because crime victimization is a rare event, lower numbers of victimizations in subsequent interviews do not necessarily indicate fatigue. Furthermore, some have argued (Cantor, 2007) that as respondents learn the NCVS instrument they answer the screener more accurately, reducing the number of erroneous victimizations reported. However, given the additional interview length that occurs when a respondent reports a victimization and decrease in the average number of incidents as interview number increases, it is highly plausible that some level of fatigue occurs.

Based on these considerations, it is likely that the NCVS is underestimating the rate of crime. Therefore, we recommend implementing an adjustment that accounts better for TSE (both telescoping error and fatigue) or, because there is uncertainty around the exact level of the adjustment, one possible option is to not apply an adjustment at all. Under either case, the resulting victimization rate is higher than the currently estimated rate. The unadjusted victimization rates were higher than the estimates with just the bounding adjustment and lower than any of the combined TSE adjusted victimization rates. This implies that the fatigue adjustment in the combined TSE adjustment was larger than the telescoping adjustment. Therefore, if there is some question about the fatigue adjustment, the unadjusted estimates may be an acceptable alternative.

Limitations

The analysis relied solely on public use data files available through the National Archive of Criminal Justice Data. Because it was necessary to link households and persons across survey years, this meant that the analysis was limited to 2007 and later because IDs were re-scrambled in 2006. In addition, because of the reliance on public use data we focused our analysis on error sources we could directly measure ourselves. This led us to focus on errors caused by the panel design. Other error sources in the TSE framework exist in the NCVS, but they were out of this paper’s scope. Furthermore, because of the complex nature of both telescoping and fatigue and the need for a 36-month reference window to support the model-based approaches it was only possible to compare 3 years of victimization rates (2011–2013). Including additional years of data could have allowed for more refined and precise models of telescoping and fatigue, which may have resulted in more conclusive trends.

Another limitation is that the proposed adjustments are done in an ad hoc manner with the post-survey adjusted public use file. Although it may be useful for survey methodologists or researchers using the data to make their own individual adjustments as needed, given the number of error sources in any survey, expecting a user to decide which adjustments to make is too onerous. Instead, it would be better if these adjustments could be made by the survey owners (e.g., BJS in the case of the NCVS) prior to the creation of the public use file. Additionally, any adjustments made should be documented to allow user identification of data alterations and the impact of those alterations on the estimates.