Measurement is not the same as reality

Decades ago, I learned to be careful about indices ranking countries. Not only is the quality of data among countries highly variable, making the rankings questionable, but the indices themselves sometimes surprise you in unexpected ways.

My first eye-opening came in my teens or early twenties. Iowa was held up as an example of superior public education at least in part based on their very high SAT scores. Having been impressed by that evidence, some months later I was disillusioned to discover that the reality was more ordinary. Most Iowans took the comparable ACT test for college application purposes. Only those seeking to go to prestigious coastal universities bothered to take the SAT exam. As a consequence, the SAT takers were in no way representative of Iowans in general. They represented Ivy League aspirant Iowans, an entirely different group and a group, not surprisingly, with superior SAT scores.

In college, I did a lot of research using World Bank information. It was in that period when I came across some WB ranking of countries based on an index of something, let's say Liberty. I have no recollection of the details but this reconstructed example suffices to illustrate the issue. Looking at the rankings, there were a lot of anomalies and overall the index did not feel compatible with what I knew from having lived in some of those countries or having had friends that had.

For example Freedom of the Press as a variable in the index, might have shown the US doing poorly but Eastern European countries (then behind the Iron Curtain) doing well. Why? Because the index valued government funding of news reporting and therefore the US rated poorly on that whereas communist East Germany rated highly. You had to get three layers deep into the index to discover these anomalies.

We likely have something similar going on right now with PISA, the international comparison of educational outcomes. The last round or two have shown high results for China but also extraordinarily high results for Shanghai as region within China. Shanghai is a huge city of near 25 million people. Statistically it is virtually impossible for such a large population to score so far from the mean of the country as a whole. Almost certainly these high results are for some subset of Shanghai students from the best schools. We don't know the truth yet. Data reported by China says one thing, the Chinese like it that Shanghai students are the best in the world, and statistical probability points to a different conclusion. Time will tell.

All this brought to mind by this article, Questions About the Global Gender Gap by Elaine Schwartz.

The World Economic Forum (WEF) tells us that Iceland leads the world in gender equality while the women in the Philippines are #7.

[snip]

To rank gender equity, the WEF uses 14 indicators. When they measure economic participation and opportunity, wage equality for similar work is one variable. Their score for education is partially based on primary and secondary enrollment. In the health component, life expectancy counts. And with political empowerment, one area is women in government.

You can see that the metrics are pretty standard and predictable.

But some of the scores appeared surprising. Do note that Iceland’s health and survival rank is 104 out of 144 countries:

[snip]

Meanwhile, the Philippines were #1…the best possible:

[snip]

However, the life expectancy of a little girl born today in Iceland is 85.3 years and a boy, 80.9. In the Philippines, the estimate is 72.9 years for women and Filipino men, 65.7.

Why then is Iceland ranked so far below the Philippines?

The answer is the WEF methodology. Because the scores are based on a male female comparison within each country, a Filipino woman fares beautifully in terms of her male counterpart.

Similarly with education, a country like Iceland has close to 100% secondary education enrollment for males and females. In the Philippines, the numbers are closer to 60%. And yet both have the same score because we are recording a male female ratio.

To put it more bluntly, because Filipinos die younger but more equally (between the sexes), the WEF views them as a more gender equal country.

Well, perhaps, but I don't think that most people would see that as a positive outcome meriting a #1 ranking.

The more time you have to invest understanding anomalies in an index, the less valuable you often discover it to be.