We imbue a number with a respect that it often does not warrant

An excellent example of how measured attributes are always a proxy for real attributes and how it is important to keep the distinction in mind. From What makes most restaurant reviews worthless by Tyler Cowen.

…a new paper appropriately titled “Demographics, Weather and Online Reviews.” The study analyzed 1.1 million online reviews of 840,000 restaurants, looking for exogenous — or external — factors in the data. In other words, they wanted to figure out what makes us like or dislike a restaurant, beside the restaurant itself.

The results can be surprising. The diners’ education levels? No effect on actual ratings. Population of the area? Again, not so much.

But reviewers consistently gave worse ratings when it was raining or snowing outside than when it was clear. And reviewers usually liked restaurants better on warm and cool days, rather than very hot or very cold ones.

In researcher Saeideh Bakhshi’s words: “The best reviews are written on sunny days between 70 and 100 degrees … a nice day can lead to a nice review. A rainy day can mean a miserable one.”

So if you are a restaurant owner, you want to have some sort of promotion to do online reviews only on nice days.

The more basic point is a fundamental one. Whenever we measure something, it is always a proxy for reality. It may be a good proxy and a useful one , but it is still a proxy. We have to know the details of how it is measured and take into account all the external factors that might affect the measure. We rarely do that. We imbue a number with a respect that it often does not warrant.

This isn't an anti-measurement screed, simply a sensible precautionary observation.

This is similar to what happened with Amazon book reviews (a starring system). Wouldn't it be great to know what the consensus of other readers was of the book that they read? Sure, that would be useful information.

The problem is that an aggregation of starred reviews is not a proxy for quality in general, much less whether it might appeal to you as an individual reader. An Amazon starred review number is usually a better proxy for the author's publicity budget, or a proxy for degree of controversy, than it is a proxy for quality.

So what you think you are measuring isn't always actually what you are measuring. What you want to measure isn't always the same as what you have been measuring.