Numbers aren't neutral

A S Paneerselvan in The Hindu



Analysing data without providing sufficient context is dangerous


An inherent challenge in journalism is to meet deadlines without compromising on quality, while sticking to the word limit. However, brevity takes a toll when it comes to reporting on surveys, indexes, and big data. Let me examine three sets of stories which were based on surveys and carried prominently by this newspaper, to understand the limits of presenting data without providing comprehensive context.

Three reports

The Annual Status of Education Report (ASER), Oxfam’s report titled ‘Reward Work, Not Wealth’, and the World Bank’s ease of doing business (EoDB) rankings have been widely reported, commented on, and editorialised. In most cases, the numbers and rankings were presented as neutral evaluations; they were not seen as data originating from institutions that have political underpinnings. Data become meaningful only when the methodology of data collection is spelt out in clear terms.

Every time I read surveys, indexes, and big data, I look for at least three basic parameters to understand the numbers: the sample size, the sample questionnaire, and the methodology. The sample size used indicates the robustness of the study, the questionnaire reveals whether there are leading questions, and the methodology reveals the rigour in the study. As a reporter, there were instances where I failed to mention these details in my resolve to stick to the word limit. Those were my mistakes.

The ASER study covering specific districts in States is about children’s schooling status. It attempts to measure children’s abilities with regard to basic reading and writing. It is a significant study as it gives us an insight into some of the problems with our educational system. However, we must be aware of the fact that these figures are restricted only to the districts in which the survey was conducted. It cannot be extrapolated as a State-wide sample, nor is it fair to rank States based on how specific districts fare in the study. A news item, “Report highlights India’s digital divide” (Jan. 19, 2018), conflated these figures.

For instance, the district surveyed in Kerala was Ernakulam, which is an urban district; in West Bengal it was South 24 Parganas, a complex district that stretches from metropolitan Kolkata to remote villages at the mouth of the Bay of Bengal. How can we compare these two districts with Odisha’s Khordha, Jharkhand’s Purbi Singhbhum and Bihar’s Muzaffarpur? It could be irresistible for a reporter, who accessed the data, to paint a larger picture based on these specific numbers. But we may not learn anything when we compare oranges and apples.

Questionable methodology

Oxfam, in the ‘Reward Work, Not Wealth’ report, used a methodology that has been questioned by many economists. Inequality is calculated on the basis of “net assets”. The economists point out that in this method, the poorest are not those living with very little resources, but young professionals who own no assets and with a high educational loan. Inequality is the elephant in the room which we cannot ignore. But Oxfam’s figures seem to mimic the huge notional loss figures put out by the Comptroller and Auditor General of India. Readers should know that Oxfam’s study has drawn its figures from disparate sources such as the Global Wealth Report by Credit Suisse, the Forbes’ billionaires list, adjusting last year’s figure using the average annual U.S. Consumer Price Index inflation rate from the U.S. Bureau of Labour Statistics, the World Bank’s household survey data, and an online survey in 10 countries.

When the World Bank announced the EoDB index last year, there was euphoria in India. However, this newspaper’s editorial “Moving up” (Nov. 2, 2017), which looked at India’s surge in the latest World Bank ranking from the 130th position to the 100th in a year, cautioned and asked the government, which has great orators in its ranks, to be a better listener. In hindsight, this position was vindicated when the World Bank’s chief economist, Paul Romer, said that he could no longer defend the integrity of changes made to the methodology and that the Bank would recalculate the national rankings of business competitiveness going back to at least four years. Readers would have appreciated the FAQ section (“Recalculating ease of doing business”, Jan. 25) that explained this controversy in some detail, had it looked at India’s ranking using the old methodology.