Using ACASI to measure gender-based violence in Ugandan primary schools

Incidences of school-related gender-based violence (SRGBV) are common in schools around the world. SRGBV can include “acts or threats of physical, sexual, or psychological violence or abuse that is based on gender stereotypes or that targets students on the basis of their sex, sexuality, or gender identities.”¹ Incidences of violence for both girls and boys can occur on the way to and from school or while at school. Three broad categories of SRGBV include bullying and non-sexual harassment, corporal punishment, and sexual violence or harassment.¹ Research, minimum standards, and monitoring frameworks, like that of the United Nations Girls Education Initiative’s (UNGEI’s) Whole School Approach² have helped establish greater awareness of the extent of SRGBV and provide concrete steps for policy makers and implementers to address SRGBV. However, the research community is still just beginning to understand both the pervasiveness of SRGBV and its psychological and academic effects on students. Existing prevalence data, intended to inform donor programming and policy, are misrepresented for three reasons:

1. SRGBV survey questions are especially sensitive. Responding can prove emotionally distressing to the respondent.³ Such sensitivity also poses ethical challenges and considerations that can make it difficult to ensure that the principle of “do no harm” is kept front and center of research studies.²

2. Social desirability bias—an interest in responding to survey questions that align with cultural values and expectations—can cause respondents to exaggerate responses that align with culturally condoned behaviors or valued experiences or to under-report stigmatized acts or experiences.⁴ As a result, data reflect experiences or personal attributes perceived to be more socially accepted or desirable rather than the respondent’s truthful experiences.

3. There are no right or wrong answers, but there are true and untrue answers. The existing literature argues that questions of a sensitive nature (i.e., questions subject to social desirability bias) threaten response accuracy and lead to systematic reporting (i.e., exaggerating socially desirable experiences and behaviors and underreporting less desirable ones). Tourangeau and Yan even argue the more sensitive the question, the greater the likelihood of misreporting.⁵

Although researchers can prepare to collect data on such sensitive topics by becoming familiar with the social norms and drivers of stigma in the context, it remains difficult to know to what degree social desirability bias will affect self-reported data. The literature on this topic (although results are mixed based on context and study design) argues that survey administration mode can address issues of misreported data caused by social desirability bias.^5,6,7,8–14 This brief contributes to the body of literature on survey administration methods for SRGBV by providing early evidence on how to measure SRGBV prevalence among children. Specifically, this research compares Audio Computer-Assisted Self-Interview (ACASI), a confidential, self-report method of data collection, with the Face-to-Face (FTF) administration method across four surveys with varying degrees of sensitivity with Primary Grade 3 pupils in Uganda. This research also includes an analysis of the factor structure and construct validity of data collected using the two administration types.

In contrast to the conventional FTF interview method, ACASI presents survey questions in an audio format through headphones—providing the respondent with true privacy. While completing the survey, respondents select their choices through the tablet without interacting with a data collector. ACASI gives the respondent privacy, which reduces social desirability bias and allows the researcher to place confidence in the quality of data collected.5 A vast amount of literature exists on the use of ACASI in collecting data from HIV-infected patients, on sexual behavior and drug use, and on intimate partner violence.^6,7,15–20 ACASI gives the respondent privacy, which reduces social desirability bias and allows the researcher to place confidence in the quality of data collected. Respondents who use ACASI tend to report their experiences differently than those using the traditional FTF method when the experiences may be sensitive, taboo, or socially undesirable.^8,21 At the time of writing this brief, no studies have tested the effectiveness of ACASI compared to FTF when administering SRGBV or school-centered surveys, especially with a population as young as 10 years of age in low- and middle-income countries. The study described in this brief addresses this gap in the literature on ACASI.

Rates of violence against children is particularly high in Uganda, with rates of physical violence among 13–17-year-old children ranging between 44% for girls and 59% for boys.²² In 2018, RTI International piloted the use of ACASI in 12 Ugandan public primary schools with Primary Class 5 (or P5; mean age  =  11.9 years) and P7 (mean age  =  13.8 years) students. Survey forms used for this study were adapted from RTI’s Survey of Student Experiences of SRGBV.²³ Under the US Agency for International Development (USAID)-funded EdData II, Data for Education Research and Programming activity, RTI developed a suite of SRGBV surveys in 2016. RTI subsequently piloted this suite of surveys under the USAID-funded Uganda Literacy and Retention Activity in 2016 and refined the surveys based on the resulting analyses for use in occasion 1 of the longitudinal study in 2018. Our large-scale ACASI study coincided with the second occasion of the longitudinal study that occurred in 2019. To ensure comparability between the ACASI and FTF administration methods, the exact same surveys were used for ACASI as those used in the longitudinal study.

Results from the 2018 ACASI pilot in 12 schools suggested that ACASI could hold promise for reducing social desirability bias and eliciting more authentic responses. In 2019, we scaled up this study to include a randomized sample of students in 40 Ugandan public primary schools with a younger population of P3 students (mean age  =  10 years) and a suite of instruments related to SRGBV and school-centered dimensions.

We visited 40 primary schools for the FTF and ACASI data collection in 2019. (Note that the survey and research protocol were approved by RTI’s Institutional Review Board and Uganda’s The AIDS Support Organization Research Ethics Committee.) These schools were preselected as the random sample of control schools under the US Agency for International Development (USAID)/Uganda Literacy Achievement and Retention Activity longitudinal study in 2018. Under this existing study, assessor teams visit the same schools every year, and the teams use the FTF method to assess students with surveys on school climate, gender attitudes, social-emotional learning (SEL), and experiences of violence. The ACASI administration included the same school climate, gender attitude, SEL, and experiences of violence surveys so that surveys with varying levels of sensitivity could be used to compare the data collected using both ACASI and FTF administration. The second round of data collection for the longitudinal study was conducted in July 2019, during which the ACASI study took place. ACASI assessor teams visited the same schools on the same day as the FTF teams. Because the 2019 data collection was the second occasion for the longitudinal study, 12 pupils in each school had already been selected the year before to take part in the FTF longitudinal study. Therefore, one day before the day of data collection, one member of the research team sampled 12 P3 students in each school out of those who were not already part of the longitudinal sample.^** Similar to the sampling procedure for the longitudinal study in 2018, this was done by forming two lines of P3 boys and P3 girls present on the day before data collection, counting the total number of boys and number of girls in each line, and dividing by 8 to calculate a jump number for each sex. That jump number was used to sample 8 boys and 8 girls in each school, which included 2 replacements per group in case caregivers or pupils themselves did not show up to the school the following day for data collection. Lines were formed randomly by the assessor teams without influence of the teacher or school administrators. This ensured that random selection occurred for both studies at each school. All interviews were conducted in the school setting, with each student placed in a private location away from other students and teachers, usually an empty classroom or another available space. The assessor worked one-on-one with each student to first obtain assent at which time each student was informed about the nature of the questions, the availability of counselors onsite, and that if they noted they were in immediate danger, it would be reported. (Each student received a letter requesting their caregiver to come to the school to provide consent for their child to participate in the survey. Only students whose caregiver provided consent participated in the survey.) After this introduction, the FTF method continued with the assessor asking each question directly to the student. With the ACASI method, the assessor instead introduced the tablet and headphones and each individual survey. The protocol did not require same-sex enumerators for this introduction. ACASI recordings were done using a female voice for all surveys. The ACASI study used the exact same surveys that were already being administered as part of the longitudinal study (FTF group); this brief focuses on the administration method rather than the surveys themselves. (The full surveys were adapted from the following surveys: School Climate, Gender Attitudes, SEL, SRGBV, available here: https://shared.rti.org/resources?type[]=283.)

A total of 442 pupils were surveyed by the FTF (longitudinal study) assessors, and 412 pupils were surveyed by the ACASI study assessors. Survey weights were applied to these data. Table 1 shows some basic demographic information on the FTF and ACASI sampled students. The demographic makeup of both samples is similar, though about 50% of ACASI students reported being orphans, whereas only 17% of FTF students were reported as being orphans. This discrepancy could be because the FTF surveys asked parents or guardians of the students about disability and orphan status, whereas ACASI surveys asked students directly about their disability and orphan status. It is possible that the young children did not understand the definition of “orphan” when responding to the question. However, based on the sampling structure as defined above, it is improbable that students randomly sampled from the same schools would have true discrepancies in orphan status. Further cognitive interviewing around these questions could clarify how young children interpreted these terms.

Research Question 1 is centered on determining whether a difference in reporting is observed between the two survey administration methods. We find that overall reporting for the school climate, gender attitudes, SEL, bullying, and corporal punishment surveys is similar between ACASI and FTF, whereas the sexual violence survey shows drastically increased reporting of experiencing sexual violence under ACASI. We summarized the data collected with each survey into key reporting metrics. The key metrics representing the percentage of responses favorable to (1) a positive school climate, (2) positive views on gender equity, or (3) a strong sense of agency and social awareness were derived for the school climate, gender attitudes, and SEL surveys, respectively. The key metrics of the experiences of violence surveys are the prevalence rates for each of the three forms of violence. Table 2 displays these key metrics for all surveys and both administration types, overall and by sex.^†† Calculations of key metrics is as follows: $percentage of favorable responses = \frac{total \# of non-missing responses that align with desired outcome}{total \# of non-missing responses}$ $prevalence= \frac{total \# of pupils that reported experiencing at least ONE act of violence}{total \# of pupils}$ Note: If a pupil did not respond to a question or selected “Don’t Know,” the question was treated as a missing response. Only non-missing responses were used in the calculation of percentage of favorable responses metrics. Missing responses did not affect the calculation of the prevalence metrics as long as the respondent answered at least one item in the survey.

The school climate, gender attitudes, and SEL surveys show similar results when comparing ACASI with FTF. This may be explained by the fact that there is likely less desire to understate the severity of an experience if that experience is not stigmatized or sensitive in nature. Interestingly, a slight increase in the percentage of favorable responses to the gender attitudes survey and a decrease in the percentage of favorable responses to the SEL survey is observed for girls under ACASI, perhaps indicating sex-specific social desirability bias at play. These findings suggest that girls may be less likely to provide authentic responses to questions around confidence, social awareness, and other dimensions of SEL if assessed FTF. The overall bullying and corporal punishment prevalence rates are also very similar for ACASI and FTF—though ceiling effects are being observed for both. We do observe an increase in the prevalence of corporal punishment for boys under ACASI compared with boys in the FTF group. The observed differences in gender attitudes, SEL, and corporal punishment reporting for specific sexes indicate that stigmatized behaviors could be differentiated by sex and highlight the importance of selecting the best method of survey administration.

The starkest findings from this study regarding a difference in reporting are related to the sexual violence survey. A drastic increase in prevalence of sexual violence is observed under ACASI for both boys and girls. About 43% of pupils who completed the sexual violence survey with the FTF administration reported experiencing an act of sexual violence at least once in the past term, whereas 77% of the ACASI group reported this experience. This substantial difference in reporting is supported by the existing literature on ACASI.^{5,6,10-12,14,24} Based on the literature, we assume that the sexual violence survey exhibits the greatest amount of misreporting because it is likely the most sensitive in this context and is likely subject to social desirability bias.²² Thus, we would expect pupils to respond more openly when they are given privacy and confidentiality through ACASI.^7,8,19,21 Figure 1 provides a closer look at reporting for each individual item within the sexual violence survey, suggesting the increase in prevalence is not driven by a single question in the survey.

Figure 1

Item-Level Responses to Sexual Violence Survey for FTF and ACASI Sampled Pupils

To address Research Question 2, we conducted exploratory factor analysis (EFA).^‡‡ EFA was conducted because prior psychometric analysis of these surveys indicated weak factor structures. Since there was weak evidence supporting the exact number of factors present within each survey, we did not deem confirmatory factor analysis as an appropriate test for this study. All EFA conducted for this study used orthogonal rotation methods. Factor analysis aims to reduce the large number of observed variables (e.g., individual survey questions) to a few interpretable latent variables, or factors (e.g., the overarching concepts we are trying to measure), that explain the maximum amount of variation in the data. Running EFA allows us to summarize findings related to the overall concepts rather than findings related to specific items. If factor structures change across different administrations of the same survey, this can mean that different constructs are being measured.

We find that the factor structures under the school climate, gender attitudes, SEL, and experiences of violence surveys for the FTF and ACASI administrations are roughly the same. (A factor loading cut-off of 0.20 was used to determine what items to drop during the EFA.) Figure 2 shows the FTF and ACASI factor structures with labels assigned according to item groupings and the associated root mean square error approximations to provide goodness of fit. (The gender attitudes and SEL surveys under ACASI and the gender attitudes survey under FTF do not meet the recommended levels for goodness of fit²⁵ [RMSEA<0.08].) Though all other surveys show the same factor structure across administration modes, the sexual violence survey under the FTF administration shows a two-factor model whereas the same survey under the ACASI administration shows a one-factor model. This may be explained by the fact that social desirability bias has been shown to affect factor structure,²⁶ making the direct comparison difficult for those surveys that may be susceptible to increased levels of social desirability bias. In fact, factor invariance tests show us that the gender attitudes, corporal punishment, and sexual violence surveys indicate at least metric invariance—factors do not largely differ across administration types. Table 3 provides results of factor invariance tests for each survey. Overall, it is evident that the type of administration does not largely affect the constructs being measured and using ACASI may not alter pupils' understanding of the items.

Figure 2

Factor Structures Present in the Data

Overall, it is evident that the type of administration does not largely affect the constructs being measured, and using ACASI may not alter pupils’ understanding of the items.

Research Question 3 focuses on construct validity, which is the degree of confidence that an identified construct is well-measured by its associated group of items. The average variance extracted (AVE) is a metric that calculates the amount of variance of a group of items that is captured by a single construct (versus the amount of variance caused by measurement error). A low AVE indicates that the construct cannot successfully capture variance coming from the individual items and therefore may not be a strong representation of those items. A high AVE indicates that the construct can capture much of the variance coming from the individual items, making it a good representation of those items. Generally, AVE values of at least 0.50 are seen as acceptable.²⁷

The AVE values for all factors in the focal tests (i.e., School Climate, Gender Attitudes, SEL, and Experiences of Violence) are smaller than the acceptable 0.50 threshold under both FTF and ACASI. The consistency of low AVE values across administration types is indicative of the tool itself not showing strong construct validity, regardless of administration type. It is important to note that items with factor loadings of at least 0.20 were included in the factor designations. Because AVE calculations are based on factor loading values, increasing this threshold to items with loadings greater than 0.30 or 0.40 would result in higher AVE values.

It is also interesting to observe that the AVE for almost all surveys increases under the ACASI administration compared with FTF, though no survey shows sufficient evidence that construct validity is supported under either administration mode. Table 4 displays the AVE for each factor in the FTF and ACASI surveys. We recommend further survey adaptation and psychometric testing to assess construct validity of these surveys under both administration methods.

The findings from this study provide early evidence, in alignment with the literature, that the ACASI administration may obtain valid and reliable data when used with children to collect sensitive, self-reported data. Factor analysis of our collected data indicates that using ACASI largely does not affect the constructs being measured, though further piloting and testing is recommended to improve construct validity.

Increased responses to the corporal punishment (for boys) and sexual violence surveys administered with ACASI indicates that ACASI can mitigate social desirability bias when collecting such sensitive data from pupils. We also observe a difference for the SEL and gender attitudes surveys when using ACASI with girls, indicating that these surveys could be subject to sex-specific social desirability bias. The findings herein are strong evidence for rejecting the FTF interview method in favor of ACASI for the administration of the experiences of violence survey. The importance of using ACASI when administering surveys of a sensitive nature is clear, but further examination regarding financial and logistical tradeoffs across methods should be considered. Also, surveys should always be piloted, and psychometric analyses should be conducted when administering surveys in new contexts or with different populations. Further methods research should also examine survey administration types that have proven to mitigate response bias because of fear, threat of disclosure, and revicitimization⁵—a bias category that was not covered under this research. Such methods would include the anonymized list survey⁹ that should be further explored vis-à-vis ACASI and FTF with pupils in similar contexts.

It is of the utmost importance that the research community take responsibility when testing human subjects by ensuring the most confidential and comfortable survey administration is used. While using ACASI, children can feel more comfortable disclosing their experiences, knowing their privacy is safeguarded. Obtaining data through proper privacy measures, which reduce the potential for bias, is the first step toward determining the full breadth and depth of SRGBV. The data can be used to better inform programming that addresses the experiences and drivers of SRGBV to eradicate violence against children in schools.

The authors sincerely thank Dr. Elizabeth Randolph, Chris Cummiskey, Geri Burkholder, Simon King, Amy Mulcahy-Dunn, and the RTI M&E Team at USAID’s Literacy Achievement and Retention Activity for their support through the original piloting of this work in 2017 and scale-up in 2019. This work is a result of a team effort. This research was funded by RTI International through an Internal Research and Development (IR&D) award.

RTI Press Associate Editor: Jeffrey Rosen