Informing the Water and Sanitation Sector Policy: Case Study of an Impact Evaluation Study of Water Supply, Sanitation and Hygiene Interventions in Rural Maharashtra, India
Pattanayak, S. K., Poulos, C., Wendland, K. M., Patil, S. R., Yang, J. C., Kwok, R. K., & Corey, C. G. (2007). Informing the Water and Sanitation Sector Policy: Case Study of an Impact Evaluation Study of Water Supply, Sanitation and Hygiene Interventions in Rural Maharashtra, India.
Sustainable and equitable access to safe water and adequate sanitation are widely acknowledged as important development goals (e.g., UN, 2005; UNDP, 2006). Yet, to date we have few or no rigorous scientific impact evaluations showing that water supply and sanitation (WSS) policies are effective in delivering many of the desired outcomes (Fewtrell and Colford, 2005). Impact evaluation measures the impacts of the program on individuals, households, and communities, and determines whether the program caused these impacts (Baker 2000; WB-OED 2004). An impact evaluation relies on control or comparison groups, as well as a number of statistical and econometric techniques to determine what would have happened in the absence of the program – this is known as the counterfactual.
We have speculated on several potential reasons for the paucity of rigorous impact evaluations in the WSS sector elsewhere (Poulos et al. 2006), but we focus here on three reasons, all of which are features of WSS policies and interventions.
First, mechanisms to achieve these goals are broad and varied in terms of the types of services (water supply, water quality, sanitation, sewerage, and hygiene); the setting (urban, peri-urban, rural); and the typology of delivery (public intervention, private interventions, decentralized delivery, expansion or rehabilitation). While these complex interventions call for carefully designed evaluation studies, most previous impact evaluations have had insufficient designs for measuring program impacts and/or for measuring the full range of impacts. One exception is the UNDP (2006) report linking adequate and safe water and sanitation to Millennium Development Goals (MDG) related to child mortality, maternal health, poverty and hunger, education, gender equity, women empowerment, combating HIV, malaria and other diseases, and environmental sustainability.
Second, decentralized and community-level projects – particularly those that are community-demand-driven (CDD) or community participation based – are an important and growing class of development projects. The combination of voluntary participation in self-selected interventions by communities and targeted provision by program administrators increases the difficulty of identifying an appropriate control group.
Third, the breadth of effects of WSS policies, which range from greater efficiency in the sector at the national level to improved health at the individual level, raises two challenges for impact evaluation. The first is that the engineering and fiscal outputs that are tracked in a Management Information System (MIS) by many projects yield little information on the effects of the program on poverty reduction – the underlying goal of development processes. The second challenge is that assessing these broad impacts requires a thorough and thoughtful approach to study design. Most impact evaluations of WSS programs focus primarily on health or a limited set of outcomes, and therefore, do not collect enough data to evaluate other impacts such as increased educational opportunities, improved rural livelihoods, or gender equity.
Most rigorous impact evaluations in the sector are quasi-experimental designs due to the fact that the features of water and sanitation programs often make randomized designs infeasible. In particular, many WSS programs are both targeted by program administrators and/or driven by community demand, making random assignment for a study inappropriate.
Our purpose in this paper is to show how quasi-experimental impact evaluation tools can be used to address these three key evaluation challenges in the WSS sector. These tools are illustrated by describing a case study of the design and implementation of a quasi-experimental evaluation of a public WSS intervention in Maharashtra, India.