By Eric Djimeu Wouabe, GAIN’s Senior Technical Specialist, Impact Evaluation
Countries and donors increasingly recognize the benefits of evaluating public programs that could increase access to safe and nutritious foods for the poor. However, low- and middle-income countries face three main challenges to evaluation. First is scale: many important programs that are good candidates for evaluation are implemented nationally. Second is perception: governments want to emphasize the effectiveness of programs they support, but evaluation can reveal shortcomings, with potential political repercussions. Third is capacity: evaluation planning and coordination can be a struggle in contexts characterized by limited specialized skills and expertise. Nonetheless, evaluation is feasible, as one recent study shows, and has potentially strong benefits for improving large-scale nutrition interventions.
In a landmark study, Bougma and colleagues (2018) took advantage of a gradual rollout of salt iodization to examine the impact of iodized salt on young children’s mental development in Ethiopia. Their study – a community-based, cluster randomized effectiveness trial – showed that iodized salt had an effect on children’s mental development scores at a time when salt iodization had not yet been mandated in Ethiopia. Iodized salt improved verbal (receptive language) and non-verbal (cognitive and fine motor) scores, as well as the composite mental development score. Effect sizes were somewhat small. This result provides a strong evidence that even brief exposure to iodized salt leads to a meaningful improvement in young children’s mental development.
The study fills an important knowledge gap. No study using a randomized design had previously been published on the effects of salt iodization on the mental development of children under 2 years. Previous evidence suffered from methodological limitations: high attrition, uncontrolled confounders and small sample size. Other studies on the effects of salt iodization on children’s mental development focused on children whose pregnant mothers received iodine supplements.
In addition to producing strong evidence, Bougma and colleagues’ study suggests three lessons about evaluating nutrition interventions in low- and middle-income countries.
Lesson 1: Important nutrition interventions can be evaluated.
One of the main challenges of evaluating nationally implemented nutrition interventions is finding appropriate comparison groups. However, financial and administrative constraints in developing countries often limit the possibility of introducing a new intervention in all communities, so most new interventions are rolled out gradually. In 2011, when the Ethiopian government passed universal salt iodization legislation, it was not feasible to serve all markets.
The need to phase in a program geographically provides an excellent opportunity to evaluate the program. Randomization could be the fairest way of helping to choose the order of phase-in. Bougma and colleagues used this approach, randomly assigning early market introduction of iodized salt in Ethiopia. A similar approach was used to evaluate the impact of the PROGRESA program on child health in Mexico.
Lesson 2: It is possible to evaluate market-based interventions supported by governments.
Governments and donors are increasingly recognizing the importance of evaluating public programs. However, the foundation of the vast majority of policies is still ideology and experience. It takes courage for a government to allow such an evaluation, and serious coordination of efforts to make the evaluation work. A gap exists between the desire to assess the effectiveness of public programs and the political economy of evaluation. Governments are sometimes reluctant to have their programs (or the programs they support) evaluated, because a lack of an effect can be perceived as a failure, with negative political repercussions. One reason for this perception is misunderstanding of what evaluation can deliver. Evaluation is not considered as a tool for improving public programs. In addition, because evaluation results are not always available for short-term decisions, they can appear difficult to translate into immediate policy actions. Finally, a successful evaluation requires advance planning and strong coordination to select an external evaluator, find funding, and possibly adjust the rollout of a program to accommodate the evaluation.
All of these conditions can be challenging in contexts characterized by limited funding and limited specialized skills and expertise. Nonetheless, this study shows it is possible to evaluate market-based interventions supported by a government – in this case, the Ethiopian government.
Lesson 3: Measurable impact for complex outcomes is achievable with a rigorous design.
Bougma and colleagues’ study shows that measurable impact for complex outcomes, such as child development, is achievable with a rigorous design, even with a minimal period of differential exposure. In this study, children in the intervention group received about 4 to 6 months more exposure to iodized salt than controls. This period of differential exposure could be insufficient to observe measurable effect, because child development represents a dynamic combination of biological and environmental factors.
Two factors can explain the measurable effect observed in this study. First, the potential for impact was high at the launch of national iodization efforts in 2011. At baseline in 2010, only 5.4 percent of households tested were using iodized salt. Thus, a significant increase of the availability of iodized salt during a short period produced measurable impact. Second, the strength of the evaluation design limited the influence of omitted variables that can bias the result of an evaluation. For example, mothers with a strong preference for early breastfeeding could be more likely to use iodized salt. A strong design, such as the one used in this study, limits the influence of this preference, which can render a measurable impact difficult to observe, especially with a short period of differential exposure.
Evaluating important nutrition interventions planned for scale at the national level is feasible.
The case of salt iodization in Ethiopia is an illustration. The evaluation of these programs requires (1) taking advantage of gradual rollout of programs; (2) an understanding of factors in the political economy that could hinder adoption of evaluation findings and strategies for overcome barriers; and (3) assessing the potential impact of these programs. Evaluation of nutrition interventions can provide improved knowledge about the types of programs that work in increasing access to safe and nutritious foods for the poor.
Published 2 May 2018