Problematic Global Metrics

Jim Church

An added benefit of doing library instruction is you learn things from students and faculty. This knowledge informs both collection development and research consultations. It is especially interesting when a new faculty member arrives and issues a revised syllabus for a popular course. One such class at UC Berkeley is in the Global Poverty and Practice (GPP) minor, founded by Professor Ananya Roy ten years ago. Her book, Poverty Capital: Microfinance and the Making of Development, makes the uncomfortable point that people and institutions profit from poverty: it is a lucrative business. But there are also those who attempt to create and influence “poverty knowledge.” The 1998 subtitle of the World Bank’s flagship publication, the World Development Report, was “Knowledge for Development.” In 2017 the World Bank wrote a feature news article (about itself) as a “knowledge institution.”1 There are articles that trace the history of the World Bank’s vision of itself as a “knowledge bank,” a term I find both amusing (do they charge “interest”?) and problematic.2 Yet a library is also a knowledge institution, and what we purchase or recommend influences the thinking and research of students and scholars.

Poverty Capital was published in 2010, and much has changed since. The primary text for the revised GPP course is The Divide: Global Inequality from Conquest to Free Markets, by Jason Hickel, which I recently purchased, read, and highly recommend. The Divide interrogates several key measures (such as the GINI Coefficient and the Millennium Development Goals) that many of us have taken at face value for years.3 In a spirit of skeptical inquiry, it is our duty as information professionals to question these metrics and encourage our users to do the same. What follows are examples of some problematic global metrics that have been challenged by scholars and recently caught my attention.

The Doing Business Index

One of the World Bank’s key publications is Doing Business. The stated purpose of this annual publication is to investigate “the regulations that enhance business activity and those that constrain it” for 190 countries.4 Each country receives an annual “DB” rank: in 2020 New Zealand placed first with a DB of 86.8; the United States was sixth, and Somalia placed last, with a DB of 20.0.5 The World Bank naturally explains its methodology: the metric is a “composite index” (typically a number from 0 to 100 comprising weighted components) and includes “starting a business,” “dealing with construction permits,” “getting credit,” and “enforcing contracts,” among others.6

Some of you may be thinking of the US News and World Report “best colleges” rankings. For as long as I can remember no public university was included in the top ten national universities: every so often a public school makes the top twenty. The “best” ones always seem to be the rich privates. It is true that there is an additional index for “global universities” where publics do better. But at the end of the day, the ranking seems to many critics little more than a proxy for wealth, fame, and SAT scores. The rankings have also, according to Cathy O’Neil (author of Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy), encouraged schools to cheat: “Bucknell University lied about SAT averages from 2006 to 2012, and Emory University sent in biased SAT scores and class ranks for at least eleven years, starting in 2000.”7

The same thing could be argued about Doing Business. Wealthy countries generally rank high: developing countries cluster toward the bottom. In the 2020 report only two countries in Africa south of the Sahara made the top fifty. No Latin American countries were included. There is again an incentive to cheat and tinker with the data: some countries actually pay the World Bank for advice on raising their rankings.8 But regardless of the methodology, how does this index help the poorest countries? As noted by Jayati Ghosh, professor of economics at Jawaharlal Nehru University, “the index has already caused huge damage to developing countries, and it should be scrapped” and “the overall thrust of Doing Business is . . . the fewer regulations a country has, the better it performs on the index.”9 Or as Isabel Ortiz and Leo Baunach observe, “A country ranks better when its social security contributions are low.”10

The index was also questioned by Paul Romer, formerly chief economist at the World Bank and 2018 recipient of the Nobel Prize in Economics. He noticed that the ranking for Chile fluctuated depending on which political party was in power (when socialist president Michelle Bachelet was elected it fell, and when she left office it rose.)11 Romer later retracted his claims, but questions about the DB methodologies remain: my favorite is from a blog post by Sandefur and Wadwha, who quip, “On almost all dimensions, a Hobbesian state of nature would get the best possible Doing Business score.”12 The criticism mounted again in 2020, which led the World Bank to issue a statement noting “A number of irregularities have been reported regarding changes to the data in the Doing Business 2018 and Doing Business 2020 reports . . . that were inconsistent with the Doing Business methodology.”13 The publication has been paused as a result.

The Corruptions Perception Index

Years ago, I presented on NGO data at a Documents Association of New Jersey conference. At that time the NGO Transparency International (TI) was making waves and I thought they were fantastic: here was an emerging nonprofit fearlessly reporting levels of global corruption. What’s not to like? Corruption—so we have been taught—is a serious issue (in developing economies) and a significant reason why development assistance fails. I will never forget the reaction of the audience when I showed TI’s signature Corruptions Perception Index (CPI) country map: several people gasped and laughed. Per Jason Hickel, this map “depicts most of the global south smeared in the stigmatizing red that paints a high level of corruption. By contrast, rich western countries, including the United States, and the United Kingdom are painted in happy yellow, suggesting very little corruption at all.”14

Whenever you evaluate metrics, look for biases. What is being measured, and according to whom? According to TI’s website, the “CPI scores and ranks countries/territories based on how corrupt a country’s public sector is perceived by experts and business executives.”15 So the CPI measures elite institutional perceptions of corruption: it does not present an objective measure.16 Not surprisingly, the CPI shows a “highly significant correlation with real gross domestic product per capita.”17 It also does not evaluate the private sector: the global financial institutions who keep getting caught assisting criminals with tax evasion and money laundering are not included. Yet the index remains widely quoted and accepted by governments, businesses, and news organizations. The University of Pennsylvania ranked TI twentieth in its 2019 Top Think Tanks Worldwide (NON-US), and in January 2020 the European Commission noted with evident pleasure that the “European Union continues to be the best performing region in the world.”18

To be clear, the CPI has received its fair share of criticism.19 TI are also not above giving developed countries a hard time: they have an online library with reports critical of business, national governments, and international organizations.20 The organization also works on other projects, including the Global Corruption Barometer (https://www.transparency.org/en/gcb), which includes a series of regional reports that measure corruption as perceived by ordinary citizens. They have helped to elevate the issue of corruption into the public consciousness. But to many the CPI remains problematic, given its exclusion of the private sector, its oversimplification of a complex problem, and its reliance on elite respondents.

Global Poverty Measures

The number of people living in poverty has been a subject of interest for decades. It is one thing, the story goes, to tell stories, take pictures, and interview people, but the best measure of poverty is data. But how does one measure poverty? How does one define it? Who is making these definitions, creating the metrics, and to what extent are they influenced by ideologies or institutional reputations?

Many of us are aware of the pervasiveness of statistical shenanigans. But until I came across discussion about this in The Divide, I did not know about the revision of the United Nations Millennium Development Goal, Target 1 (MDG-1), which was to halve world poverty. As Hickel relates with a mixture of anger and flair, the first version was drafted in the Millennium Declaration adopted at the conclusion of the UN Millennium Summit in September 2000.21 This declaration is a General Assembly Resolution (A/RES/55/2) and states the world will aspire to

Halve, by the year 2015, the proportion of the world’s people whose income is less than one dollar a day.22

But the version published in the 2008 UN Millennium Development Goals Report states

Halve, between 1990 and 2015, the proportion of people whose income is less than $1 a day.23

The original version makes no mention of the year 1990. The UN backtracked the baseline ten years, taking credit for a decade of growth (mostly in China) before the MDGs were even implemented (facepalm). These metrics are also proportions (not absolute numbers) thus taking advantage of the higher population growth in the developing world: for more information, see the discussions by Hickel and Pogge.24

Even more fraught is the debate about the World Bank’s International Poverty Line (IPL). The current version, which the Bank uses to count the number of people living in “extreme poverty” worldwide, was revised in October 2015 and stands at $1.90 a day. How is this amount arrived at? According to the Bank’s most recent report, it is based on the “national poverty lines of the same countries that previously defined the $1.25 line” in 2008. These are the fifteen poorest countries in the world, whose poverty lines are set by their national governments.25 As Pogge notes, “It is unclear why political decisions made by rulers or bureaucrats in a few poor countries should be thought a reliable indicator of what ‘poverty’ means to poor people all over the world.”26 Naturally, this leads to significant discrepancies between the poverty counts reported by other national governments. To be fair, the World Bank includes the alternative poverty baselines of $3.20 and $5.50 a day PPP (as well as national government levels) in its famous World Development Indicators database.27 If you change the measure you get very different poverty counts. But the most widely quoted figures are based on the IPL.

The UN Food and Agriculture Organization’s (FAO) State of Food Insecurity in the World provides another disquieting example. In 2009, the FAO reported the number of hungry people worldwide at 1.02 billion people, which made it difficult for the UN to claim it was on the road to halving poverty and hunger by 2015.28 But in 2012 the FAO also changed its methodology, stating that “improved undernourishment estimates, from 1990, suggest that progress in reducing hunger has been more pronounced than previously believed” and that “the revised results imply that the Millennium Development Goal (MDG) target of halving the prevalence of undernourishment in the developing world by 2015 is within reach.”29 The new headcount thus stood at 870 million—appalling, but still a reduction. But as Hickel elaborates, the revised measure is only valid if you calculate hunger based on the caloric needs of a sedentary lifestyle—hardly appropriate for those engaged in the manual labor prevalent in poor countries. If the FAO were to calculate the levels for a “normal” lifestyle, in 2012 between 1.5 billion and 2.5 people would have been hungry.30

I am not an international statistician. But I am reasonably numerate, and these methodologies seem problematic. Skepticism is always in order when examining and quoting important metrics that governments use to create policy. I am also not claiming that all the tremendous work put in by IGO and NGO employees to alleviate poverty was in vain, or that no progress has been achieved. Many of the gains accomplished during the Millennium campaign and other initiatives are tremendous accomplishments, and should be justly celebrated. What I am saying is we always need to question the methods used to collect statistics, and to teach this skill to our users.

Jim Church (jchurch@library.berkeley.edu), University of California, Berkeley

Notes

1. World Bank, “The World Bank—a Knowledge Institution,” www.worldbank.org/en/news/feature/2017/05/03/the-world-bank-provider-of-knowledge.
2. Teresa Kramarz and Bessma Momani, “The World Bank as Knowledge Bank: Analyzing the Limits of a Legitimate Global Knowledge Actor,” Review of Policy Research 30, no. 4 (2013): 409–31, https://doi.org/10.1111/ropr.12028.
3. The Gini coefficient is an indicator of income inequality ranging from zero to one: zero indicates perfect equality, one perfect inequality. For more information on the eight UN Millennium Development Goals (MDGs) see https://www.un.org/millenniumgoals/.
4. World Bank, Doing Business 2020, https://www.doingbusiness.org/en/reports/global-reports/doing-business-2020.
5. World Bank, Doing Business, 4.
6. World Bank, Doing Business, 17.
7. Cathy O’Neil, “How Big Data Makes Applying to College Tougher, Crueler, and Ever More Expensive,” Slate Magazine, September 15, 2016, https://slate.com/business/2016/09/how-big-data-made-applying-to-college-tougher-crueler-and-more-expensive.html.
8. According to The Economist. See “The World Bank’s Business-Rankings Mess,” The Economist, September 3, 2020, https://www.economist.com/finance-and-economics/2020/09/03/the-world-banks-business-rankings-mess.
9. Jayati Ghosh, “Stop Doing Business,” Project Syndicate, September 10, 2020, https://www.project-syndicate.org/commentary/world-bank-should-scrap-doing-business-index-by-jayati-ghosh-2020-09.
10. Isabel Ortiz and Leo Baunach, “It Is Time to End the Controversial World Bank’s Doing Business Report,” International Development Economics Associates (IDEAs), September 3, 2020, https://www.networkideas.org/news-analysis/2020/09/it-is-time-to-end-the-controversial-world-banks-doing-business-report/.
11. Josh Zumbrun and Ian Talley, “World Bank Unfairly Influenced Its Own Competitiveness Rankings,” Wall Street Journal (online), January 12, 2018.
12. Justin Sandefur and Divyanshi Wadhwa, “Chart of the Week #3: Why the World Bank Should Ditch the ‘Doing Business’ Rankings—in One Embarrassing Chart,” Center for Global Development, January 18, 2018, https://www.cgdev.org/blog/chart-week-3-why-world-bank-should-ditch-doing-business-rankings-one-embarrassing-chart.
13. World Bank, “Doing Business—Data Irregularities Statement,” August 27, 2020. https://www.worldbank.org/en/news/statement/2020/08/27/doing-business---data-irregularities-statement.
14. Jason Hickel, The Divide: Global Inequality from Conquest to Free Markets, First American edition (New York: Norton, 2018), 209.
15. Transparency International, “Corruption Perceptions Index,” https://www.transparency.org/en/cpi.
16. These are listed at https://images.transparencycdn.org/images/2019_CPI_SourceDescription_EN-converted-merged.pdf and include, among others, the World Economic Forum, the EIU, the World Bank, Freedom House, and HIS Global Insight.
17. Paul G. Wilhelm, “International Validation of the Corruption Perceptions Index: Implications for Business Ethics and Entrepreneurship Education,” Journal of Business Ethics 35, no. 3 (February 1, 2002): 177, https://doi.org/10.1023/A:1013882225402.
18. James McGann, “2019 Global Go to Think Tank Index Report,” June 18, 2020, p. 61, https://repository.upenn.edu/think_tanks/17/; European Commission, “The Corruption Perception Index 2019: The EU Is the Best Performer in the World,” January 27, 2020, https://ec.europa.eu/regional_policy/en/newsroom/news/2020/01/27-01-2020-the-corruption-perception-index-2019-the-eu-is-the-best-performer-in-the-world.
19. Stuart Campbell, “Perception Is Not Reality: The FCPA, Brazil, and the Mismeasurement of Corruption,” SSRN Scholarly Paper, February 21, 2013, https://papers.ssrn.com/abstract=2210019.
20. See, for example, https://www.transparency.org/en/publications/governance-international-maritime-organisation and https://www.transparency.org/en/publications/g20-leaders-or-laggards.
21. Hickel, The Divide, 36–39.
22. United Nations, General Assembly, Fifty-fifth session, 2000, United Nations General Assembly Resolution 55/2, United Nations Millennium Declaration, UN Doc. Symbol A/RES/55/2, https://undocs.org/A/RES/55/2, p. 5.
23. United Nations, “Millennium Development Goals Report 2008,” https://www.un.org/millenniumgoals/2008highlevel/pdf/newsroom/mdg%20reports/MDG_Report_2008_ENGLISH.pdf, p. 6.
24. Hickel, The Divide, 37; Thomas Pogge, “How World Poverty is Measured and Tracked,” in Elke Mack, Stephan Klasen, and Thomas Pogge, eds. Absolute Poverty and Global Justice: Empirical Data, Moral Theories, Initiatives. Law, Ethics and Economics (Farnham, England; Burlington, Vt.: Ashgate, 2009), 51–65.
25. World Bank, World Bank 2015 Global Monitoring Report, 2015, pp. 5 and 10, http://pubdocs.worldbank.org/en/109701443800596288/PRN03Oct2015TwinGoals.pdf.
26. Pogge, “How World Poverty is Measured,” 56.
27. PPP = Purchasing Power Parity, a measure to calculate the actual “purchasing power” of different countries’ currencies.
28. Food and Agriculture Organization of the United Nations, The State of Food Insecurity in the World, 2009, p. 2, http://www.fao.org/3/i0876e/i0876e00.htm.
29. Food and Agriculture Organization of the United Nations, The State of Food Insecurity in the World, 2012, http://www.fao.org/3/a-i3027e.pdf. See introduction (“key message”).
30. Hickel, The Divide, 45.