Skip to Main Content

RTI uses cookies to offer you the best experience online. By and clicking “accept” on this website, you opt in and you agree to the use of cookies. If you would like to know more about how RTI uses cookies and how to manage them please view our Privacy Policy here. You can “opt out” or change your mind by visiting: http://optout.aboutads.info/. Click “accept” to agree.

Accept
RTI International
  • About
    • Office Locations
    • Executive Leadership
    • Corporate Governance
    • Partner with Us
      • U.S. Government
      • Clients and Funding Agencies
      • Industry and Commercial Clients
      • Foundations and Associations
      • Bilateral Agencies and Multilateral Banks
      • Universities and Academic Research Institutions
      • Suppliers and Small Businesses
    • Commitment to Quality
      • RTI's Client Listening Program
    • Ethics and Human Research Protection
    • Living Our Mission
    • Veteran Opportunities at RTI

    About

  • Practice Areas
    • Health
      • Public Health and Well-Being
      • Health Care Transformation
      • Behavioral Health
      • Health Behavior Change
      • Precision Medicine
      • RTI Health Solutions (RTI-HS)
      • RTI Center for Community Health Evaluation and Economics Research
      • Health Equity
      • RTI Health Advance
    • Transformative Research Unit for Equity​
      • Equity Capacity Building Hub
      • Social and Economic Justice Research Collaborative
      • Narrative Research and Community Engagement Lab
    • Education and Workforce Development
      • Early Childhood
      • K-12 Education
      • Postsecondary Education
      • Career and Adult Education and Workforce Development
      • Education Policy, Systems, and Governance
      • Education Research Methodologies
      • Education Technologies
    • International Development
      • Energy for Development
      • Environment
      • Global Food Security, Agriculture, and Nutrition
      • Global Health
      • International Education
      • Monitoring, Evaluation, Research, Learning, and Adapting (MERLA)
      • Youth and Economic Opportunity
      • Building Resilience Against COVID-19 in Developing Countries
      • Water, Sanitation, and Hygiene (WASH)
      • RTI Center for Governance
    • Climate Change
      • Clean Energy Technology and Renewables
      • Climate Finance
      • Climate Justice and Equity
      • Climate Planning, Preparedness and Resilience
      • Climate Policy
      • Climate Vulnerability, Adaptation, and Mitigation
      • Economic Impacts of Climate Change
    • Water
      • Food-Energy-Water Nexus
      • Water Quality
      • WASH (Water, Sanitation, Hygiene)
      • Water Resources Management
    • Energy Research
      • Carbon Capture and Utilization
      • Biomass Conversion
      • Natural Gas
      • Energy Efficiency
      • Industrial Water
      • Syngas Processing
    • Environmental Sciences
      • Air Quality
      • RTI Center for Water Resources
      • Urban Sustainability
      • Toxics
      • Building Resiliency in the FEW Nexus
      • Climate Change Sciences and Analysis
      • Environmental Policy
      • Environmental Justice
      • Sustainable Materials & Waste Management Solutions
    • Justice Research and Policy
      • RTI Center for Community Safety and Crime Prevention
      • RTI Center for Policing Research and Investigative Science
      • Child Well-Being and Family Strengthening
      • RTI Center for Forensic Sciences
    • Food Security and Agriculture
      • Market Systems Strengthening
      • Food Safety
      • Food and Nutrition
      • Global Food Security, Agriculture, and Nutrition
      • Climate-Smart Agriculture
      • Agricultural Innovation
      • Obesity Prevention
    • Innovation Ecosystems
      • Innovation Advising
      • Innovation for Economic Growth
      • Innovation for Emerging and Developing Economies
      • Innovation for Organizations
      • Research, Technology, and Innovation Policy
      • Technology Acceleration
    • Military Support
      • Military Behavioral Health
      • Military Health and Human Performance
      • Military Sexual Assault, Harassment, and Domestic Violence Prevention
      • Wearable Sensor Technologies
      • Military Health System Transformation

    Practice Areas

  • Services + Capabilities
    • Surveys and Data Collection
      • Survey Design
      • Instrument Development
      • Survey Methodologies
      • Data Collection
      • Establishment Surveys
      • Health Registries
      • Data Analysis and Reporting
      • Research Operations Center
    • Statistics and Data Science
      • Survey Statistics
      • Environmental Statistics
      • Coordinating Centers for Multisite Studies
      • Analysis and Design of Complex Data
      • Biostatistics
      • RTI Center for Data Science
    • Evaluation, Assessment and Analysis
      • Evaluation Design and Execution
      • Advanced Qualitative, Quantitative, and Mixed Methods
      • Evaluation, Monitoring, and Assessment
      • Economic Analysis
      • Evaluating Communication Interventions and Campaigns
      • Evidence Synthesis for Policy and Practice
      • Risk Assessment and Prediction
    • Program Design and Implementation
      • Systems Strengthening and Scaling
      • Capacity Assessment and Building
      • Policy Reform Support
      • Curriculum and Teacher Professional Development
      • Interventions and Prevention Programs
      • Implementation Science
    • Digital Solutions for Social Impact
      • Human-Centered Design of Digital Solutions
      • Digital Product Development
      • Digital Communication Campaigns
      • Digital Data Analytics
    • Research Technologies
      • Survey Technologies
      • Data Management and Decision Support Systems
      • Geospatial Science, Technology, and Visualization
      • ICT for Limited-Resource Settings
      • Mobile Applications
      • Web Applications
      • Bioinformatics
      • Interactive Computing
    • Drug Discovery and Development
      • Medicinal Chemistry
      • Molecular Design and Cheminformatics
      • Behavioral Pharmacology
      • Drug Metabolism and Pharmacokinetics (DMPK)
      • In Vitro Pharmacology, Bioassay Development, and High-Throughput Screening (HTS)
      • Isotope Labeling
      • Regulatory Consulting and Support for Medical Products
    • Analytical Laboratory Sciences
      • Bioanalytical and Toxicology Research
      • Forensic Sciences
      • Physicochemical Characterizations
      • Metabolomics
      • Proficiency Testing and Reference Materials
      • Microbiology
      • Analytical Chemistry and Pharmaceutics
    • Engineering & Technology R&D
      • Biomedical Technologies
      • Decarbonization Sciences
      • Environmental Exposure & Protection
      • Materials & Environment
      • Sustainable Energy Solutions

    Services + Capabilities

  • Centers
    • RTI Center for Advanced Methods Development
    • RTI Center for Communication Science
      • Communication Research
      • Communication Design
      • Communication Delivery
    • RTI Center for Data Science
    • RTI Center for Education Services
      • Teaching and Learning
      • Education Leadership
      • Peer Learning Networks
      • Strategic Consulting
    • RTI Center for Forensic Sciences
    • RTI Center for Global Noncommunicable Diseases
      • Program Financing & Economics for NCDs
      • Health Systems Strengthening for NCDs
      • Communication Science and Behavior Change for NCDs
      • Implementation Science for NCDs
    • RTI GenOmics, Bioinformatics, and Translational Research Center
      • Disability Studies
      • Ethics
      • Newborn Screening
    • RTI Center for Water Resources
      • Water Resources Sectors
      • Water Resources Services
      • Water Resources Tools
    • RTI Center for Governance
    • RTI Global Gender Center
    • North Carolina Center for Optimizing Military Performance
    • NCCU-RTI Center for Applied Research in Environmental Sciences
    • RTI Center for Climate Solutions

    Centers

  • Impact
    • Newsroom
    • Insights Blog
    • Events
    • Publications
    • RTI Press
      • About the RTI Press
      • Instructions for Authors
      • RTI Press Collections
    • Projects
    • Global Reach
      • Asia
      • Eastern Europe and Central Asia
      • RTI International India
      • Africa
      • Middle East and North Africa (MENA)
      • Latin America and the Caribbean (LAC)

    Impact

  • Experts
    • Our Experts
    • In-Depth With Our Experts
    • Related News
    • Experts In the Media
    • RTI Fellow Program

    Experts

  • Emerging Issues
    • COVID-19 Research
    • Artificial Intelligence
    • Global Health Security
    • Cannabis Research
    • Opioid Research
      • Interventions for Opioid Use Disorders
      • Preventing Opioid Misuse and Overdose
      • Treating Opioid Use Disorders
    • Policing Research and Investigative Science
    • Drone Research and Application
    • E-cigarette Research
    • Zika Virus Research
    • Integrated Governance

    Emerging Issues

  • COVID-19 Research + Response
  • Global Reach
  • Insights Blog
  • Newsroom
  • RTI Press
  • Publications
  • Partner With Us
  • Careers
  • Facebook IconTwitter IconInstagram IconYouTube IconLinkedin Icon
  • Home
  • Impact
  • RTI Press
  • Agreements between public health organizations and food and beverage companies

Agreements between public health organizations and food and beverage companies

Approaches to improving evaluation

By Jean L. Wiecha, Mary K. Muth.

January 2021 Open Access Peer Reviewed

DOI: 10.3768/rtipress.2021.op.0067.2101

Check for Updates Download PDF
Wiecha, J. L., & Muth, M. K. (2021). Agreements between public health organizations and food and beverage companies: Approaches to improving evaluation. RTI Press. RTI Press Occasional Paper No. OP-0067-2101 https://doi.org/10.3768/rtipress.2021.op.0067.2101
Copy citation
Share
  • Share on Facebook
  • Share on Twitter
  • Share on Linkedin
  • Email
Key Points
  • Food and beverage companies often enter into voluntary industry agreements as a form of industry self-regulation to improve the healthfulness of foods and beverages and potentially avoid government regulation.
  • Increasing scientific scope and rigor in evaluating voluntary industry agreements would improve potential public health benefits and understanding of the effects of these agreements.
  • Evaluators must provide objective, data-driven input from the outset of the planning process through the end of the project.
  • Logic models, industry profiles, quasi-experimental designs, mixed-methods approaches, and third-party data can improve the effectiveness of agreement design and evaluation.
  • Improved access to data and larger evaluation budgets will be needed to support improvements in evaluation.

Abstract

Efforts in the United States and abroad to address the chronic disease epidemic
have led to the emergence of voluntary industry agreements as a substitute for
regulatory approaches to improve the healthfulness of foods and beverages. Because of the lack of access to data and limited budgets, evaluations of these
agreements have often been limited to process evaluation with less focus on outcomes and impact. Increasing scientific scope and rigor in evaluating voluntary food and beverage industry agreements would improve potential public health benefits and understanding of the effects of these agreements. We describe how evaluators can provide formative, process, and outcome assessment and discuss challenges and opportunities for impact assessment. We explain how logic models, industry profiles, quasi-experimental designs, mixed-methods approaches, and third-party data can improve the effectiveness of agreement design and evaluation. These methods could result in more comprehensive and rigorous evaluation of voluntary industry agreements, thus providing data to bolster the public health impacts of future agreements. However, improved access to data and larger evaluation budgets will be needed to support improvements in evaluation.

Creative Commons © 2023 RTI International. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Contents

  • Introduction
  • Developmental Evaluation
  • Process, Outcome, and Impact Evaluations
    • Process Evaluation
      • Outcome Evaluation
        • Impact Evaluation
        • Data Sources and Uses
        • Execution Decisions
        • Conclusion
        • Acknowledgments
        • References

        Agreements Between Public Health Organizations and Food and Beverage Companies: Approaches to Improving Evaluation

        By Jean L.WiechaMary K.Muth

        Introduction

        Substantial research implicates poor diet as a major factor contributing to noncommunicable diseases like diabetes, cardiovascular disease, stroke, and some types of cancer (US Burden of Disease 2017 Diet Collaborators, 2019; Mokdad et al., 2018). The food and beverage supply chain, including commodity suppliers, manufacturers, distributors, and retail, has a dominant role in improving food and beverage healthfulness in the marketplace through reformulating or introducing new products and changing retail, pricing, and promotion practices. One strategy for encouraging changes in the food and beverage supply chain is developing voluntary industry agreements between industry and public health partners (e.g., nongovernmental organizations [NGOs]). Recent examples in the United States include the Healthy Weight Commitment, in which 16 of the leading consumer packaged food and beverage manufacturers committed to selling 1.5 trillion fewer calories from 2007 through 2012 (Mozaffarian, 2014; Ng & Popkin, 2014; Ng, Slining, & Popkin, 2014), and the Balance Calories Initiative, in which the top three beverage companies pledged to reduce calories consumed from sugar-sweetened beverages by 20 percent from 2015 through 2025 (Cohen et al., 2018).

        Voluntary industry agreements promote specific changes in product formulation, marketing, or distribution negotiated between industry and a public health partner. As such, they usually require participation in monitoring and evaluation. These agreements can be considered a type of industry self-regulation, but transparency, meaningful benchmarks, oversight, and objective evaluation are needed to ensure that they have a true public health impact (Sharma, Teret, & Brownell, 2010). Guiding principles developed for establishing agreements and partnerships that promote healthy food environments cite the need for accountability through monitoring and evaluation (Kraak, 2014; Kraak & Story, 2015). In addition, Kreslake and Sawyer (2017) developed a series of recommendations for evaluating voluntary industry agreements that called for more standardization and rigor in design, more focus on outcome measurement, and improvements in measurement quality.

        Understanding the potential public health benefits of voluntary industry agreements has often been limited because most evaluations of these agreements focus on process evaluation involving activity verification. In a related paper, we assessed 20 peer-reviewed review articles and commentaries on voluntary industry agreements and 15 peer-reviewed evaluations of voluntary industry agreements (Wiecha & Muth, 2021). This assessment indicated that industry partners have a strong influence on agreement designs, which often weakens public health aims. Process evaluations, which are typically all that can be accomplished given evaluation budgets, have shown that some commitments have been implemented with fidelity to plans (e.g., reduction in sugary beverage calories shipped to schools, as noted in Wescott, Fitzpatrick, & Phillips [2012]). However, others have shown limited changes, like stocking vending machines accessible to primary school students with healthy foods (Royo-Bordonada & Martínez-Huedo, 2014). Overall, most evaluations were conducted using designs that limited the understanding of attribution and causality of effects of the agreements on public health outcomes.

        Interviews with 15 food and beverage companies, evaluation firms, and foundations that fund evaluations reinforced the fact that although industry agreements were usually driven by specific public health concerns, evaluation typically focused on implementation rather than public health outcomes (Wiecha & Muth, 2021). In addition, third-party evaluation lends credibility to industry agreements, but limited access to data, limited budgets, and industry partner influence reduce its effectiveness (Wiecha & Muth, 2021).

        This paper builds on the findings in Wiecha and Muth (2021), Kreslake and Sawyer (2017), and Kraak (2014) to provide guidance on approaches and methods for improving the evaluation of voluntary industry agreements. Although we focus on evaluation of voluntary industry agreements that involve a public health organization working with a company to establish an agreement, most of the methods are also applicable to evaluations by researchers of initiatives that companies pursue independently. Our recommended steps provide information to public health researchers that could help strengthen agreement design and improve evaluation objectivity and rigor. This information is also useful to public health organizations that broker industry agreements and therefore must make decisions about evaluation goals as well as to the organizations that fund evaluations and therefore must make decisions about budgets to allocate for evaluation.

        Developmental Evaluation

        Evaluators should be engaged from the planning phases of each agreement and moving forward. In developmental evaluation, an evaluator helps with project design, focusing on developing appropriate goals and objectives. Engaging evaluators is particularly useful in dynamic, adaptive, and complex settings like the food industry. Developmental evaluation can help ensure that the agreement will meet partners’ needs and have public health value and that the implementation plan is appropriately sized and achievable. Furthermore, it can identify potential unintended consequences and stimulate thinking on mitigation strategies. Developmental evaluation is typically an iterative process; it includes background research and structured, ongoing communication with goals of building strategy based on real-time information. In this way, stakeholders—NGO partners, businesses, funders, and expert advisors—can identify and adapt efficient solutions in different contexts (Preskill & Beer, 2012). Additional stakeholders include government agencies and foundations that focus on public health goals.

        Table 1 lists examples of questions that developmental evaluation activities could answer. Two outputs that can address these questions are industry profiles and logic models. Industry profile reports are based on publicly available data. They are most useful if completed before developing a contractual commitment but would still be useful if developed later because they can provide a 360-degree look at the industry in which a potential business partner operates. Industry profiles typically describe (1) the supply side of the industry (i.e., manufacturers, distributors, retailers, and foodservice), (2) the demand side (i.e., consumers as individuals or households, including demographic information and knowledge, attitudes, and behaviors), and (3) external factors affecting the industry such as regulations and general trends.

        Questions that can guide design and evaluation of industry initiatives
        Phase Guiding Questions
        Developmental Evaluation Setting:
        •      What industries and companies have strong potential for meaningful action and partnerships?
        •      Which are ready for engagement and change?
        Goals:
        •      What are reasonable end points for a collaboration?
        •      Is the aim formative (e.g., establishing trust among partners or implementing a pilot), outcome oriented (e.g., improving availability of healthful products), or impact oriented (e.g., improving dietary quality)?
        •      What science-based nutrition, health, health equity, and environmental goals are partners interested in addressing or achieving?
        •      What are the business goals?
        Outcomes:
        •      What activities or agreement elements are appropriate for the goals and are feasible and measurable?
        Evaluation Planning •      What is the perceived evaluability of the initiative based on whether data are available or could be collected to measure the intended outcomes?
        •      Is there interest in and funding for conducting a process evaluation, outcome evaluation, or both?
        •      Is there interest in and funding for developing qualitative, quantitative, or a combination of measures?
        •      What are existing data sources than can be used for the evaluation, and what new data collection is required?
        •      Will the design permit evaluators to attribute improvements in the intended outcomes to the initiative?
        •      What structures are in place to protect evaluator independence and objectivity?
        •      What are the potential unintended consequences of the initiative, and can they be assessed in the evaluation?
        Process Evaluation •      Did the initiative achieve its objectives, and was it implemented with fidelity to design?
        •      To what extent was it implemented as planned, and in what contexts or settings did the best implementation fidelity occur?
        •      What was the industry partner’s perspective on feasibility, acceptability, and fidelity during the implementation?
        Outcome Evaluation •      What effect did implementation have on availability and accessibility of healthier food and beverage options in the marketplace?
        •      What effect did implementation have on marketing and promotion strategies, events, and imprints?
        •      What effect did implementation have on consumer purchase and consumption behavior?
        •      Were there any adverse consequences, such as compensatory marketing of unhealthy products in other markets or environmental impacts?
        Impact Evaluation •      What effect did implementation have on intake of the targeted foods and beverages?
        •      What effect did implementation have on diet quality of the target population?
        •      What effect did implementation have on weight status of the target population, and are the changes sustainable?
        •      What effect did implementation have on reducing the incidence of noncommunicable diseases such as diabetes and cardiovascular disease in the target population?

        A more in-depth profile could capture information that will help determine if a potential corporate partner is a good fit for a voluntary industry agreement—that is, whether the partner has the potential to truly ally with the public health partner’s goals. In addition to describing the structure of the industry, evaluators can use profiles to estimate the potential outcomes of an industry initiative in terms of the relative market shares of the affected products, businesses, or stages of the supply chain. Industry initiatives that affect greater market shares may have more potential public health impact, but they may also carry larger risks to the business partner because they will likely have larger impacts on business operations. The profile can increase transparency during the partnership-building process by describing corporate social responsibility activities, core product healthfulness, areas of corporate self-interest, and industry or company practices that undermine population or environmental health. For example, corporate social responsibility activities may demonstrate intent to improve health, whereas other corporate activities may appear to promote unhealthy foods or beverages. The toolkit provided in Kraak (2014) for assessing partnership opportunities and challenges can also be used to help guide the development of an industry profile. The Access to Nutrition Index-US, which scored a selection of multinational food companies on health-related dimensions of corporate policy, may also be an additional relevant source when developing an industry profile (Access to Nutrition Foundation, 2018). The factors used to rate companies for the Access to Nutrition Index-US—such as information on corporate strategy, production formulation and nutrition labeling commitments, product and brand promotion practices, and product accessibility (Sacks et al., 2019)—could provide relevant context.

        The second output is a heuristic consisting of a logic model and a theory of change for a specific initiative. If possible, these should be informed by the industry profile. The logic model should clearly identify inputs, activities, outputs, and anticipated outcomes and impacts (if any) of the agreement (US Department of Health and Human Services, Centers for Disease Control and Prevention, Office of the Director, Office of Strategy and Innovation, 2011). A simple visual representation and a more complex version with additional descriptive language may both be necessary to ensure clarity and foster consensus. Although logic models help visualize what will happen under ideal circumstances—that is, a causal sequence—a theory of change addresses how. It describes the context and assumptions, establishes the causal pathway from interventions to outcomes, and identifies challenges or potential bottlenecks that could be addressed proactively (Taplin & Clark, 2012). The evaluator should work closely with the public health partners and participating companies to develop the logic model and theory of change that best represent the agreement at the outset, and the team should use these documents to devise their workplans. Going forward, they should revisit and revise them as circumstances dictate.

        Process, Outcome, and Impact Evaluations

        During the developmental phase, industry and public health partners reach consensus on the specific activities and goals of an agreement. Next, the evaluator can work with them to identify the resources available for evaluation, including budget and data, and proceed toward developing an evaluation design and plan. Partners should adopt the evaluation plan long before implementation begins to permit baseline data collection. Establishing data availability and evaluator access is of primary importance. In this section, we define process, outcome, and impact evaluations in the context of voluntary industry initiatives and subsequently discuss design, data collection, and data sources.

        Process Evaluation

        All voluntary industry initiatives should include a process evaluation to assess the extent to which objectives were met (e.g., all retail locations posted informational displays, changes in product formulation or pricing were made). Questions addressed in a process evaluation, as shown in Table 1, can assess how much implementation occurred and under what circumstances, as well as fidelity, feasibility, and acceptability on multiple dimensions, including cost, to help with decisions about scaling and sustainability. Process evaluations can focus on outputs listed in the logic model, framed as measurable, time limited, and, if possible, quantitative objectives. Supplemental qualitative information can shed light on implementation processes, successes, and challenges to enable quality improvement and scaling. Because outcomes cannot be attributed to an intervention that was not implemented, process evaluations are necessary for effective attribution.

        In many industry initiatives, process evaluation has been the final measurement step, often because of limited data access or insufficient evaluation budgets. In some circumstances, a terminal process evaluation may be sufficient; it may help a new industry partner feel comfortable with the initiative and with the evaluation process, it may confirm feasibility, or it may provide quality improvement data. An important role of a terminal process evaluation is to examine questions about implementation. Not all agreements are structured to achieve significant changes in the marketplace, and those that have more formative goals such as proof of concept, feasibility testing, and trust-building can still have value. Measuring these end points could clarify whether agreements have benefits related to increasing entry points into the healthy food movement for industry partners who are new to the table.

        Outcome Evaluation

        The focus on process evaluation and limited outcomes measurement has contributed to criticism that voluntary industry agreements lack evidence of effectiveness. Outcome evaluations assess what occurred as a result of implementation and illuminate whether activities led to healthful changes in the marketplace and in consumer attitudes and choices. Outcome evaluations may rely on a mix of quantitative and qualitative data (e.g., document reviews, environmental scans, and interviews) that respond to questions such as those listed in Table 1.

        Outcome evaluations require baseline and follow-up data that include sales, marketing, and distribution data. Industry partners may have some of this information if they choose to make it available to evaluators, but they may have restrictions on the level of proprietary information they can share. Other data sources exist, including using sales and marketing data that can be purchased from commercial data companies; conducting primary data collection, including surveys of consumers and businesses and environmental scans (i.e., broadcast media time sampling data); and obtaining government data. These datasets each have some limitations but with appropriate caveats, their use would expand knowledge about how voluntary industry initiatives affect the marketplace. We discuss their use below under Data Sources and Uses.

        Impact Evaluation

        Impact in health research typically refers to health-related effects of outcomes. In theory, if population reach for voluntary industry initiatives is sufficient, population-level surveillance data that measure diet quality, health status, and weight status could be used to identify improvements over time. Impact evaluations typically require rigorous designs that address change over time or change relative to a comparison group. However, attribution to specific industry initiatives will likely remain difficult, if not impossible, because of the numerous other factors affecting producer and consumer behavior over time. Although there is a low likelihood that a single voluntary industry initiative would merit impact evaluation, we list potential impact evaluation questions in Table 1.

        Data Sources and Uses

        Many evaluations of industry initiatives rely on data from the industry partner to assess implementation. However, other data sources, such as primary data collection or purchased scanner and other sales data, are alternatives, but typical evaluation budgets will need to increase to cover the costs. Table 2 lists possible measurement methods that can be used for evaluating industry initiatives stratified by main sources of data (e.g., company- or organization-provided data, collected data, purchased data, government data) and indicates the potential applicability of each measurement method to process, outcome, and impact evaluation. The types of organizations that could enter into an industry initiative include manufacturers, grocery retailers, restaurants, and other entities that offer foods and beverages for purchase or consumption. Thus, the specific data that can be obtained from records, collected through primary means, and purchased will vary and affect what can be measured. Companies may be reluctant to share extensive data with external evaluators because of concerns that it could compromise their competitive position, in which case primary and purchased data may be preferred alternatives, assuming the evaluation budget is sufficient. Analysis methods will vary depending on the nature of the data, ranging from descriptive summaries of qualitative data, to tabulations and statistical testing of quantitative measures, to regression-based modeling approaches. As noted in Table 2, government surveillance data can be used for impact evaluation but in most cases, the anticipated effects of industry initiatives are not expected to be large enough to detect measurable change in specific geographic areas or nationally.

        Possible measures and data sources for evaluation of industry initiatives
        Source of Data Measurement Methods Applicability of Measurement Method
        Process Evaluation (or Verification) Outcome Evaluationa Impact Evaluationb
        Company-provided records
        (e.g., from manufacturers, stores, restaurants, vending machines)
        Product formulation, packaging, and labeling records
        Menu offerings and nutrition information
        Product advertising, including paid and owned media (media type, volume, expenditures)
        Product shipment records (destination, quantity, attributes)
        Product purchase records (source, quantity, attributes)
        Product sales records (quantity, attributes, prices)
        Store planograms
        Employee health and wellness program documentation
        Website content (initiatives, education, calculators)
        Organization-provided records (e.g., from schools, day care, community-based organizations) Product purchase records (source, quantity, attributes)
        Menu records
        School policy documentation
        Employee health and wellness program documentation
        Website content (initiatives, education, calculators)
        Federal program participation records
        Primary data collection Site visits (schools, NGOs)
        In-depth stakeholder interviews
        Consumer focus groups
        Consumer surveys (telephone, mail, intercept, social media, mail, opt-in web panel, nationally representative web panel)
        Industry surveys (multiple modes)
        Organization surveys (multiple modes)
        Store and restaurant audits (offerings, pricing, signage)
        Commercial data (available for purchase) Store scanner data (sales, prices)
        Household scanner data (purchases, prices; with demographics)
        Label data (nutrition facts, ingredients, claims)
        Consumer food purchase diary data (foods consumed, food description, source of food; with demographics)
        Menu data (offerings, prices, nutrition)
        Product advertising, including paid, owned, and earned media data (media type, volume)
        Government data Consumer Expenditures Survey (purchase value)
        NHANES dietary recall (consumption)
        NHANES anthropometric measures (height, weight)
        NHANES survey (attitudes, behaviors, food spending)
        BRFSS (self-reported height, self-reported weight, food frequency)
        NHIS (self-reported height, self-reported weight, health status)
        YRBSS (self-reported height, self-reported weight, food frequency)
        BRFSS = Behavioral Risk Factor Surveillance System; NHANES = National Health and Nutrition Examination Survey; NHIS = National Health Interview Survey; YRBSS = Youth Risk Behavior Surveillance System.
        Outcome measures relate to awareness, understanding, attitudes, availability, affordability, accessibility, reformulation, labeling, marketing, pricing, purchases, and consumption.
        Impact measures relate to health status and weight status.

        Execution Decisions

        Some of the decisions that evaluators must consider in executing an evaluation are shown in Table 3. Most industry agreements are not structured to accommodate experimental designs, but quasi-experimental designs can yield evidence of cause and effect if appropriately planned with respect to selection of end points, sample size, data quality, data collection, and data analysis. Well-executed time-series analyses that include sufficient baseline data are generally the best option available to evaluate implementation and outcomes of voluntary industry agreements. Both time-series and pre-post designs with comparison groups will provide better evidence than pre-post designs without comparison groups or post-only designs. An evaluator may want to identify the strengths and weaknesses of different levels of approaches (e.g., offering bronze, silver, and gold standard options) that fit the goals and funding for the evaluation and discuss how to optimize design selection with business and public health partners.

        Evaluation design and execution decisions
        Type of Decision Examples
        Sampling strategy Selection of localities, businesses, products, households, or other observational units for assessing impacts
        Timing Time period and frequency of data collection relative to the initiative’s timeline, including capturing baseline data before initiation
        Comparisons Pre-post, time-series, or control group comparisons
        Triangulation Whether results can be triangulated across multiple sources to strengthen validity and provide qualitative context in interpreting quantitative findings

        Conclusion

        Evaluators can help ensure that voluntary industry initiatives for improving food and beverage healthfulness are operating as intended, having the desired effect on food industry practices and leading to meaningful outcomes. To do this, evaluators must provide objective, data-driven input from the outset of the planning process through the end of the project. Developmental input can include helping public health partners identify suitable industry partners and set measurable, realistic, and meaningful objectives for implementation and outcomes. Doing so helps ensure that public health partners support agreements that can have a true public health impact and avoid those that merely provide companies with talking points about their engagement in public health initiatives. Subsequently, evaluators can design appropriate process and outcome evaluations that measure progress and assess barriers and facilitators to change; measure implementation; and assess actual outcomes and impact on sales and consumer knowledge, attitudes, and behavior. In addition to ensuring that voluntary industry agreements are operating as intended, more comprehensive and rigorous evaluation could provide information to inform the development of future agreements to improve public health impact. However, improved access to data and larger evaluation budgets will be needed to facilitate these efforts.

        Acknowledgments

        Support for this research was provided by the Robert Wood Johnson Foundation (Grant No. 74483). The views expressed here do not necessarily reflect the views of the Foundation. We also acknowledge the advice and support of Priya Gandhi, Victoria Brown, and Tina Kauh at the Foundation.

        RTI Press Associate Editor: Jules Payne

        References

        Access to Nutrition Foundation. (2018, November). Access to nutrition index. U.S. spotlight index 2018. https://accesstonutrition.org/app/uploads/2020/02/Spotlight_Index_US-Index_Full_Report_2018.pdf
        Cohen, D. A., Bogart, L., Castro, G., Rossi, A. D., Williamson, S., & Han, B. (2018). Beverage marketing in retail outlets and The Balance Calories Initiative. Preventive Medicine, 115, 1–7. 10.1016/j.ypmed.2018.07.01430055198
        Kraak, V. (2014). Public health and food and beverage industry engagement: A tool to assess partnership opportunities and challenges. The Healthy People and Communities Steering Committee's Multi-Sectoral Partnerships Task Group. https://www.paho.org/hq/dmdocuments/2015/ppptg-guide.PDF
        Kraak, V. I., & Story, M. (2015). Guiding principles and a decision-making framework for stakeholders pursuing healthy food environments. Health Affairs, 34(11), 1972–1978. 10.1377/hlthaff.2015.063526526257
        Kreslake, J., & Sawyer, R. (2017). Study of private sector practices to address childhood obesity and overweight in the United States, 2005–2015. FHI 360.
        Mokdad, A. H., Ballestros, K., Echko, M., Glenn, S., Olsen, H. E., Mullany, E., Lee, A., Khan, A. R., Ahmadi, A., Ferrari, A. J., Kasaeian, A., Werdecker, A., Carter, A., Zipkin, B., Sartorius, B., Serdar, B., Sykes, B. L., Troeger, C., Fitzmaurice, C., Rehm, C. D., Santomauro, D., et al. US Burden of Disease Collaborators. (2018). The state of US health, 1990–2016. Burden of diseases, injuries, and risk factors among US states. JAMA, 319(14), 1444–1472. 10.1001/jama.2018.015829634829
        Mozaffarian, D. (2014). The Healthy Weight Commitment Foundation trillion calorie pledge: Lessons from a marketing ploy? American Journal of Preventive Medicine, 47(4), e9–e10. 10.1016/j.amepre.2014.07.02925240970
        Ng, S. W., & Popkin, B. M. (2014). The Healthy Weight Commitment Foundation pledge: Calories purchased by U.S. households with children, 2000-2012. American Journal of Preventive Medicine, 47(4), 520–530. 10.1016/j.amepre.2014.05.03025240968
        Ng, S. W., Slining, M. M., & Popkin, B. M. (2014). The Healthy Weight Commitment Foundation pledge: Calories sold from U.S. consumer packaged goods, 2007-2012. American Journal of Preventive Medicine, 47(4), 508–519. 10.1016/j.amepre.2014.05.02925240967
        Preskill, H., & Beer, T. (2012). Evaluating social innovations. 10.22163/fteval.2012.119
        Royo-Bordonada, M. A., & Martínez-Huedo, M. A. (2014). Evaluation of compliance with the self-regulation agreement of the food and drink vending machine sector in primary schools in Madrid, Spain, in 2008. Gaceta Sanitaria, 28(1), 65–68. 10.1016/j.gaceta.2013.05.00823849854
        Sacks, G., Vanderlee, L., Robinson, E., Vandevijvere, S., Cameron, A. J., Ni Mhurchu, C., et al. Swinburn, B. (2019). BIA-Obesity (Business Impact Assessment-Obesity and population-level nutrition): A tool and process to assess food company policies and commitments related to obesity prevention and population nutrition at the national level. Obesity Reviews, 20(Suppl2), 78–89. 10.1111/obr.1287831317645
        Sharma, L. L., Teret, S. P., & Brownell, K. D. (2010). The food industry and self-regulation: Standards to promote success and to avoid public health failures. American Journal of Public Health, 100(2), 240–246. 10.2105/AJPH.2009.16096020019306
        Taplin, D. H., & Clark, H. (2012). Theory of change basics: A primary on theory of change. ActKnowledge.
        US Burden of Disease 2017 Diet Collaborators. (2019, May 11). Health effects of dietary risks in 195 countries, 1990–2017: A systematic analysis for the Global Burden of Disease Study 2017. Lancet, 393(10184), 1958–1972. 10.1016/S0140-6736(19)30041-8
        US Department of Health and Human Services, Centers for Disease Control and Prevention, Office of the Director, Office of Strategy and Innovation. (2011). Introduction to program evaluation for public health programs: A self-study guide. Centers for Disease Control and Prevention.
        Wescott, R. F., Fitzpatrick, B. M., & Phillips, E. (2012). Industry self-regulation to improve student health: Quantifying changes in beverage shipments to schools. American Journal of Public Health, 102(10), 1928–1935. 10.2105/AJPH.2011.30061022897528
        Wiecha, J. L., & Muth, M. K. (2021). Better evaluation can improve the effectiveness of agreements between public health organizations and food and beverage companies. American Journal of Evaluation. 10.1177/1098214020979782

        Click cover to download publication

        Keep Exploring

        • icon-externallink-blue Created with Sketch.

          Related paper in the American Journal of Evaluation.

        • icon-externallink-blue Created with Sketch.

          HTML version [Scholastica]

        Sustainable Development Goals

        • Goal 2: Zero Hunger
        • Goal 3: Good Health and Well Being

        Contact

        To contact an author or seek permission to use copyrighted content, contact our editorial team

        • +1 919 541 6490
        • rtipress@rti.org

        Meet the Experts

        View All Experts

        Mary K. Muth

        Related Publications

        View All Press
        OCCASIONAL PAPER

        Bringing an equity-centered framework to research

        OCCASIONAL PAPER

        Culturally informed community engagement

        RESEARCH REPORT

        Substance misuse prevention program attendance

        RESEARCH BRIEF

        COVID-19’s impact on clinical research

        OCCASIONAL PAPER

        Social determinants of health

        OCCASIONAL PAPER

        Challenges facing CAHPS surveys and opportunities for modernization

        RESEARCH REPORT

        Artificially intelligent social risk adjustment

        OCCASIONAL PAPER

        The need for a diverse environmental justice workforce

        Recent Publications

        View All Press
        OCCASIONAL PAPER

        Bringing an equity-centered framework to research

        OCCASIONAL PAPER

        The Preschool Entitlement

        OCCASIONAL PAPER

        Culturally informed community engagement

        RESEARCH REPORT

        Substance misuse prevention program attendance

        RTI Logo
        Partner With Us
        • US Government
        • Commercial
        • Foundations & Associations
        • Multilateral Donors
        • Universities
        • Suppliers
        Site
        • Privacy Policy
        • Security Policy
        • Site Map
        • Terms of Use
        • Accessibility
        • Contact Us
        Contact Us
        Facebook Icon Twitter Icon Instagram Icon YouTube Icon Linkedin Icon
        delivering the promise of science
        for global good
        RTI Health Solutions RTI Innovation Advisors RTI Health Advance

        © 2023 RTI International. RTI International is a trade name of Research Triangle Institute. RTI and the RTI logo are U.S. registered trademarks of Research Triangle Institute.