The Intersection of Evaluation and Innovation

[caption id="attachment_9211" align="aligncenter" width="610"] Schoolkids in Rwanda use computers distributed by American NGO One Laptop per Child. The initiative has come in for criticism over price, impact, and educational priorities.
Photo by One Laptop per Child, via Flickr. Creative Commons.[/caption]

In light of the ongoing process to craft the post-2015 development agenda and Sustainable Development Goals, it is imperative that the international community takes the right approach towards development in Africa. Realizing large-scale goals such as ending poverty, achieving gender equality, and ensuring access to energy for all is extremely difficult, especially given the cross-cutting nature of these issues and the number of actors involved. The world has spent decades and billions of dollars working towards these lofty goals through aid programs and loans, yet has consistently fallen short.

Part of the challenge is that achieving goals like ending poverty or ensuring universal education requires institutions and processes that take time to build and are often difficult to influence externally. The failure of the World Bank's structural adjustment loans clearly illustrates this point. However, there is much that international aid can do to improve the lives of the poorest populations in Africa. Ensuring that aid is effective and makes a tangible difference requires rigorous evaluation and feedback mechanisms, both of which are often the weakest aspect of development programs.

International aid continues to throw good money at bad projects — investing where the impact is often unmeasured, marginal, or even negative. Many international development programs do not include robust feedback mechanisms that measure what works and what does not, mainly because organizations are loath to spend precious funding on monitoring and evaluation, rather than on tangible goods. Furthermore, organizations often lack the capacity and necessary training to carry out strong evaluations. All donors, organizations, and agencies need to concentrate more resources on short and long-term monitoring and evaluation to inform future programs. Rigorous evaluation should be the standard, not the exception. However, even when proper monitoring and evaluation does occur, the results rarely influence program strategy going forward.

While international development aid cannot always speed up the process of institution building or force good governance, implementing agencies can ensure that money is allocated to the right programs to achieve the best possible results. This is what the Abdul Latif Jameel Poverty Action Lab (J-Pal for short) seeks to do through randomized control tests (RCTs). Randomized evaluation is a rigorous impact evaluation method that randomly assigns resources, programs, or policies to different groups to measure the direction and size of the impact of different actions compared to a control group. For example, J-Pal tested the impact of different methods for increasing education in Kenya. They found that traditional approaches, such as hiring more teachers, or providing school meals, uniforms, or scholarships resulted in one to three more years of total schooling per USD $100 spent. However, spending USD $100 on deworming school children led to 28.6 more total years of schooling, and informing people about the benefits of education yielded 40 total years of schooling per USD $100 spent.

In light of this information, would the international community not be more effective by focusing aid for education on deworming children and educating families about the benefits of education? While these strategies could achieve the best results, international aid programs do not focus on these approaches. For example, the most recent education-themed USAID program in Kenya focused on training teachers to improve the quality of education. While it is no doubt important to train educators and ensure high-quality instruction in both developing and developed countries, recent research and evaluations, such as J-Pal's RCTs, do not suggest that this is the most productive way for USAID to spend its funding in Kenya to achieve international development goals. Even when donor and implementing agencies have impact evaluation information available, they often neglect to use this feedback to determine future funding priorities and strategies. The structure, mindset, culture, and politics of development organizations create a tendency to stick with the status quo, to the detriment of aid recipients.

While RCTs can provide valuable information about what programs work, they cannot tell us why these programs work, which is precisely what is needed to create productive international development policy. This is the biggest flaw of RCTs, according to Angus Deaton. He further points out that RCTs are costly and difficult to implement, can never be completely randomized, and are easily influence by exogenous factors. He argues, "the analysis of projects needs to be refocused towards the investigation of potentially generalizable mechanisms that explain why and in what contexts projects can be expected to work."

The key then, is to make use of all available information and select the correct evaluation methods and monitoring techniques appropriate to the situation. RCTs have their contribution to make to our understanding of development solutions, but they do not replace other methods of monitoring and evaluating each program. A strong emphasis on context-specific evaluation is perhaps most important when implementing new development products and ideas. Individuals and organizations around the world are using technology and creativity to produce innovative approaches to development challenges. While these new initiatives will not work everywhere, they do offer the potential for targeted, sustainable solutions when proper evaluation and monitoring is carried out to determine the appropriate contexts and approaches for success.

Take, for example, the problem of access to clean water. The LifeStraw allows people to drink clean water directly from a contaminated water source and lasts three to five years. The Hippo Roller, already being used in 21 African countries, makes it easier and faster for people to transport water over long distances. These two products address two different challenges that different populations face with regards to safe drinking water and the appropriateness of each product is highly dependent on the environment. The LifeStraw is valuable for people who live close to an unclean water source, while the Hippo Roller benefits individuals who live far from a clean water source. Stringent evaluation of the impact of these and other innovative products in the hands of the world's poorest people in varying contexts could equip the international community with incredible new tools for addressing development.

When testing new ideas and products, it is critical to consider the context and environment in which they are successful. Take, for example, the PlayPump, a playground merry-go-round invented in South Africa that uses the energy of children playing to pump and store water. The PlayPump received $16.4 million in donations in 2006 at a Clinton Global Initiative ceremony after early success at a few schools in South Africa. It was touted as a successful and innovative alternative to pumping water by hand, and over 1,500 PlayPumps were installed in South Africa and Zimbabwe. However, just a few years later it became clear that many of the installed PlayPumps never worked properly because they were installed in communities where there were not enough children to use them or the water was not of a high enough quality for drinking. Another fourth of the PlayPumps were nonoperational and were never repaired, cutting off access to a local water source completely.

Even as these issues came to light, PlayPumps International continued to install more and more PlayPumps in an effort to spend its donations and make use of all the pumps that had been manufactured. This is a classic, albeit perhaps an extreme, example of the failure to evaluate the impact of development solutions over time and consider the context and circumstances of their implementation, as well as a failure to respond to feedback and adjust accordingly. Africa is a diverse continent, and just because an approach works in one village or one country does not mean it will be universally successful, although this is often assumed to be true. These are common pitfalls for international development programs that can be remedied by dedicating more resources to context-specific testing, evaluation, and feedback mechanisms.

There is enormous potential for poverty alleviation and development through innovative and inexpensive ideas that are simple yet able to deliver profound results. However, it is imperative that these ideas are tested, evaluated, improved upon, and framed within the correct context and environment for success. This should be the focus of international aid — at the intersection of innovation and evaluation. By supporting the development and growth of innovative ideas, investing in rigorous monitoring and evaluation through appropriate methodology, and using the results to inform decisions about development programs, the international community can ensure that money is spent on the most effective programs to achieve the best results for international development.

Margo Berends is a program associate at Global Communities. She was previously a program intern with the Wilson Center Africa Program.