Acronym Stew: Why Some New Ingredients in International Development are Best Left Uncombined
By Ruth Naylor, CfBT Education Trust.
Popular ingredients and methods in international development come in and out of fashion, just as they do in gastronomy. As new ingredients and methods are developed and discovered, they can add richness, versatility, and efficiency savings to the work of the development professional… or to the professional cook. But as every good cook knows, you can sometimes have too much of a good thing, and even Heston Blumenthal would agree that some ingredients are best kept separate.
In the field of international development in education, Value for Money (VfM), Payment by Results (PBR), Randomised Control Trials (RCT), Early Grade Reading Assessments (EGRA) and are the current ingredients of choice. But should they be all be added, indiscriminately, to all education development projects? And what happens to these ingredients when they are combined?
VfM is an approach to development that looks at getting the maximum results for the best cost (see NORRAG NEWS 47 on this topic). PBR is a good method for achieving VfM, but only if the result being paid for is the same as the result that a project was designed to achieve.
RCT is a scientific methodology for testing a hypothesis. The result is knowledge, evidence to inform future practice. The RCT methodology is an approach designed to rule out bias. In order to conduct an RCT well, those conducting it need to make every effort of avoid bias: they need to be equally open to a negative result as a positive result. This becomes difficult if those conducting the trial have a vested interest in gaining a positive result (see for comparison Ben Goldacre’s critique of pharma’s involvement in RCTs for new medicines). Attention to avoid bias is not just confined to the data collection methodology and analysis, but needs to be included in the design, implementation and dissemination of the trial also.
If combined with PBR and the result against which payments are being made is knowledge, then that should work fine. But if the result against which payments are made is the same as the outcome that the RCT is investigating (for example, number of children able to read, number of girls attending school) there is an inherent conflict of interest for the implementing agency: should they favour unbiased furthering of knowledge or should they do everything in their means to maximise the results that they will be paid for? In this case it would be somewhat naive to expect the implementation of the RCT element to be conducted in a truly unbiased manner, even with the most principled of implementing agencies.
RCTs are VfM if the result you are after is knowledge of what works. But if the result you are after is getting girls into secondary school, or improving learning outcomes on a large scale, then an RTC is a very expensive means of evaluating the process. When it comes to delivery of education development support, the VfM approach is to apply the learning from previous RCTs, rather than to apply the RCT methodology to every new project and programme.
EGRA has become a popular method for measuring outcomes in education. However, evaluating effectiveness of interventions using EGRA only works if the intended outcome of the intervention is to improve early grade reading. If the aim is to improve social cohesion or girls’ access to secondary school, or making education more inclusive, it’s not a particularly useful tool.
The whole set of these fashionable acronyms has been incorporated into DFID’s Girls’ Education Challenge (GEC) – the aim of which is given as “to help up to a million of the world’s poorest girls improve their lives through education”. The step change funding window was about scaling up proven methodologies for getting marginalised girls through a full course of primary and secondary education. In response to this, many organisations have drawn on evidence from research to design projects aimed at increasing marginalised girls’ access to secondary education. These GEC projects are now faced with PBR on RCT findings using EGRA methodology. Is this VfM? Or have we lost our way in the acronym stew?
Ruth Naylor is a Senior International Consultant, International Development and Education at the CfBT Education Trust. Email: rnaylor@cfbt.com