What Is There Not to Like About a Global Learning Metric?

This NORRAG Highlights published by William C. Smith, Teaching Fellow in Comparative Education and International Development at the University of Edinburgh, and Aaron Benavot, Professor of Global Education Policy in the School of Education at the University at Albany-SUNY, debates how the use of GLMs (global learning metrics) effectively narrows the notion of good quality education and sidetracks concerns with equity. The authors highlight that the current approach of measuring learning globally leaves children out of school behind, entailing the danger of creating a less educated underclass.

Learning has become central to international policy discussions in education. The new global goal on education, SDG 4, prioritizes a broad learning agenda, including instruction in diverse curricular domains and school subjects; acquiring relevant skills and competencies for work and life; and exposure to values, attitudes and behaviour relevant for global citizenship and sustainability. And yet, in the process of converting this broad agenda into global and thematic indicators and, most importantly, into concrete measurement strategies, the earlier negotiated consensus about the role of learning in the 2030 education vision is breaking down. Prioritization is occurring, if not by design, then by default. Significant resources are being allocated to measure relatively few targets, while other languish “under development”.

In what appears increasingly like a zero-sum game, efforts to measure SDG 4 indicators on early childhood, functional literacy, skills for work, and knowledge and skills for education for sustainable development (ESD) and global citizenship education (GCED) are being side-lined, while attention and resources are channelled into constructing a universal learning scale of reading and mathematics proficiency at three levels – the early grades of primary education, the end of primary education and the end of lower secondary education. (This constitutes the global indicator for SDG target 4.1). Creating a global learning metric (GLM) that ranks countries on the above indicator means combining results from diverse cross-national assessments, in intricate statistical ways, in order to report the percentage of young people in each country who meet minimum proficiency standards in reading and mathematics in primary and secondary education.

Concerns about the value of global scales of learning have been around for some time. Consider, for example, the following excerpts from the Final Document of 1989 World Conference on Education for All in Jomtien, Thailand.

“One key issue…is the minimum common level of learning that must be achieved by . . . learners. A difficult aspect…will be testing the learning level or performance of individual learners…” (pp. 13-14)

“While the emphasis on learning acquisition was welcomed, some interventions cautioned against too utilitarian an approach to defining ‘an acceptable level of learning. The development of the creative potential of the individual, of imagination, of spiritual and aesthetic values, of community spirit, are justifiable in their own right, and not easily measurable in the short term …” (p. 14)

 “The point about learning is that it is a process of growth, and not a product to be acquired: learning is a journey, not a destination.” (p.14) (1)

Today ambivalence and doubts around the measurement of learning in GLMs continue unabated. Some raise questions as to whether narrowly defined GLMs undermine the comprehensive and interdependent nature of education targets set out in SDG 4 and other SDGs. Others raise concerns about the legitimacy of promoting “Western-biased” metrics of learning when education systems suffer from dilapidated schools, insufficiently prepared teachers, inappropriate instructional materials and the like. Here we focus on how the use of GLMs effectively narrows the notion of good quality education and sidetracks concerns with equity.

Prioritizing learning in the SDG 4 agenda is one thing; developing international comparable data on learning in GLMs is quite another. Will measures of learning only rely on data produced in international and regional assessments, and thereby omit data generated by national assessments? Will they only include data from school-based assessments or also integrate estimated learning levels among out-of-school children and youth? Decisions on these questions have implications far beyond the concerns of psychometricians.

Thus far, extensive time and resources have been allocated to measuring SDG 4.1.1, partly with the aim of turning this GLM into a “lead indicator” for the education sector. Led by UIS, with contributions by GAML members, these efforts have achieved some dividends: 4.1.1 is the only global indicator among the 11 assigned to SDG 4 to advance up the IAEG-SDG tier ranking system (from tier 3 to tier 2). This work would not have been possible without the substantial support of the World Bank, the Global Partnership for Education, the Hewlett Foundation, and the UK Department for International Development, which is contributing £4,600,000 (2018-2022) toward UIS work in developing and expanding the measurement of 4.1.1 and other learning-related indicators.

Early conversations around a GLM for target 4.1 highlighted the value of including results from national assessments and making use of broader surveys, including household surveys, to assess learning levels of out of school children and youth. For instance, Albert Motivans of UIS (2014) emphasized the importance of strengthening local systems so they can produce more robust learning assessments for a GLM, and better inform policy development.

However, with pressures to create a viable GLM looming on the horizon, expediency eclipsed longer term considerations.

• In 2017, UIS stated that it would not support the establishment of a new international testing platform, but rather take advantage of a wave of international assessments scheduled for 2018 and 2019.
• Several months later, it became clear that a newly emergent GLM on reading and mathematics would mainly draw on cross-national assessments and provide footnotes when necessary to detail differences in definitions of minimum proficiency.
• Throughout 2018 the UIS Director stated unequivocally that countries ought to invest resources to participate in regional and international assessments, including justifying why half a million dollars every four years is a good choice for national governments in developing countries.
• By the end of 2018 a cost benefit analysis presented three of the most likely approaches to constructing a GLM in reading and mathematics. Two of the three would not use national assessments, while the third could include national tests following the validation of a global reporting scale.

From a measurement viewpoint, creating a GLM based solely on standardized data from a discrete number of expanding international and regional assessments makes sense. From a political viewpoint, such a decision would be untenable, given widespread country concerns and sensitivities on learning issues.

Whichever measurement strategy is eventually agreed upon, it is worth reflecting on the likely consequences of such a GLM. We briefly discuss the implications of two recent attempts – one by the World Bank, and the other by UIS – that capture learning on a global scale. In January 2018, the World Bank updated their global dataset on education quality. Covering 163 countries, the dataset was created using the statistical recalibration approach, one of the three covered by the cost-benefit analysis. It relies on scores from international and regional assessments, using countries that have participated in multiple assessments to compare difficulties and scales across assessments. National assessments are not included. The harmonized scores from this dataset are then incorporated as one of five components in the World Bank’s new Human Capital Index.

Using international and regional assessments to rank countries on learning league tables raises multiple equity concerns. Countries scoring at or near the bottom of the international ranking are often heavily criticized. The pressure on education systems by donors to participate in comparative assessments translates into pressure on schools and teachers to perform well. Rather than provide constructive information to improve policy and practice, scores on such assessments are increasingly perceived as high stakes. This can lead some countries to withhold uncomplimentary results, such as China did in 2009, or argue for adjustments in score calculations, as Thailand did recently when it threatened to cease participation if the ministry was not allowed to proof read the translated questions and remove small rural schools from the sample. Results are often used to reinforce existing policies or push for reforms that mainly prompt higher future scores. Examples include narrowing official curricular policy to focus on tested subjects and question types; asking poor performing students to skip the assessment; or attempting to ‘emulate’ practices believed to be prevalent in high-performing countries.

In reality the World Bank’s new dataset, and others like it, lead to broad conclusions on learning and quality from a limited range of learning domains. Assessments privilege a few academic subjects – namely language, mathematics, and social and natural sciences – while omitting others. In a similar vein, equating quality with school-based achievement scores overlooks learning that happens outside of the school setting, contradicting the broader sense in the SDGs that learning can happen formally, non-formally, and informally throughout the life course. Not only is the learning of those outside of school not valued but because it is based on assessments administered in school, there is no account of out of school children and youth in the dataset.

Another example of a GLM in the making is the one introduced by UIS in 2017 to capture young people ‘not learning’ in the world. UIS reported that 617 million children and adolescents worldwide were not achieving minimum proficiency in reading and mathematics, a “stylized fact” that contributes to advocacy around the ‘global learning crisis’. It also provides a “solution” to the out of school issue by including figures for out of school children in the total number of not learning. By definition, out of school children are ‘not learning’ since it is assumed that all such young people lack sufficient skills in reading and mathematics. In contrast to the World Bank dataset, however, figures were not disaggregated by country (only by region), making it difficult to discern the consequences of producing simple, stylized numbers to address country challenges and donor requests.

These recent attempts by the World Bank and UIS conceive of learning in a highly specific manner: the only learning that counts is school-mediated, book-based academic knowledge. Learning outside of school doesn’t count.

If a GLM in reading and mathematics proficiency garners most public attention and donor support, what happens to the still staggering number of 263 million children worldwide who remain out of school? This question is especially pertinent in Sub-Saharan Africa, where one in five of the regions’ primary age children are out of school – a rate twice as high as the next region (Northern Africa and Western Asia). Overall aid to education was down 2% from 2016 to 2017 and has been effectively flat since 2009. The share of aid allocated to basic education in low-income countries in 2016 was the lowest since numbers have become available. As a result, while half of the world’s out of school children reside in Sub-Saharan Africa, the region only collects 25% of aid in education.

In the expensive race to construct a narrowly defined GLM, these most marginalized children and youth are likely to be forgotten. Donors’ emphasis on simple, proxy indicators of ‘quality’ and shifting global financial support away from basic education for those most in need contradicts the necessary push to get the last 9% of the world’s school age population into school. Given the higher per capita cost of getting the 90^th to 100^th percentile into school, this effort that will take additional funding.

Measuring learning globally is effectively leaving many children behind which results in undermining their ability to find employment, access technology, live healthier lives and identify as global citizens. In essence, ignoring those with minimal or no access to education will create a less educated underclass.

The march toward forgetting nearly 10% of the world’s school age children can be, in part, slowed down by reverting to earlier ideas to operationalize a GLM by including household surveys and citizen-led assessments (which capture out of school children) and incorporating national assessments and focusing on national capacity building. Pressing forward in the current direction has not only significant equity concerns but fails to consider warnings from some of the major proponents of a global metric. The Learning Metric Task Force concluded that “setting one-size-fits-all global standards is unlikely to be useful” while the World Bank cautioned “when a single metric becomes the sole basis for big policy triggers, the corresponding stakes become dangerously high” (World Bank, 2017, p. 93).

Driving any decisions in the creation of a GLM should be a set of questions reflecting potential equity issues: will the obsession over measuring and monitoring learning diminish the importance of, or effectively marginalize, equity concerns in SDG4? Will GLMs undermine support and funding for the 263 million children and youth who are excluded from school? And will country efforts to improve foundational skills in literacy and numeracy undercut innovative policies and practices to ensure that all out of school children and youth gain access to education and enjoy its benefits?

Endnote:

(1) Final Document of the 1989 World Conference on Education for All. Paris: UNESCO.

Authors’ Acknowledgements: This blog draws significantly from two forthcoming chapters:

Benavot, A., & Smith, W. C. (2019). Overlooking equity: Global learning metrics as a ready-made solution to a manufactured crisis. In Grading Goal Four: Tensions, Threats and Opportunities in the Sustainable Development Goal on Quality Education. Leiden: Brill International.

Smith, W.C. (2019). One indicator to rule them all: How SDG 4.1.1 dominates the conversation and what it means for the most marginalized. In Wiseman, A.W. (Ed.), Annual Review of Comparative and International Education. United Kingdom: Emerald Publishing.

About the authors:

William C. Smith is a Teaching Fellow in Comparative Education and International Development at the University of Edinburgh. He was previously a Senior Policy Analyst for UNESCO’s Global Education Monitoring Report. His publications on education policy and international development include his edited book The Global Testing Culture: Shaping Education Policy, Perceptions, and Practice (2016, Symposium Books). Email: w.smith@ed.ac.uk Twitter: @william_c_smith

Aaron Benavot is Professor of Global Education Policy in the School of Education at the University at Albany-SUNY. His scholarship has explored diverse educational issues from comparative, global and critical perspectives. During the 2014-2017 period Aaron served as Director of the Global Education Monitoring Report, an independent, evidence-based annual report published by UNESCO, which monitors global education trends and analyzes progress towards international education targets in the 2030 Agenda for Sustainable Development. Aaron has also co-authored or co-edited five books, including PISA, Power, and Policy (with H-D Meyer) and School Knowledge for the Masses (with J. Meyer and Kamens). Twitter: @BenavotAA

Contribute: The NORRAG Blog provides a platform for debate and ideas exchange for education stakeholders. Therefore if you would like to contribute to the discussion by writing your own blog post please visit our dedicated contribute page for detailed instructions on how to submit.

Disclaimer: NORRAG’s blog offers a space for dialogue about issues, research and opinion on education and development. The views and factual claims made in NORRAG posts are the responsibility of their authors and are not necessarily representative of NORRAG’s opinion, policy or activities.