Goal clarity and proxy measure selection

A few days ago in Mexico as an explanation of charter schools, I discussed how the Mexican experience regarding private and public schools might shed some light on an apparent paradox in the US regarding charter schools. The data from Mexico indicate that public and private schools return roughly the same average test scores as one another. This experience is similar to that here in the US between public and charter schools. Hence the paradox. From simply a data perspective, why are people so enthusiastic about charter schools when their average scores are about the same as those in public schools.

There are a number of reasonable hypotheses but the Mexican data suggests one I have not seen mooted before - Risk Aversion. In that original post, there is a chart of the bell curves for both public and private schools showing that they both have the same mean score. However, what is also apparent, is that private schools have a much tighter standard deviation than the public schools. All of the worst schools were public even though the means were the same.

This is not necessarily surprising. Private schools are subject to the discipline of the market place in terms of transparency, accountability, free choice and consequences. These elements are frequently absent or weaker in government based systems. It is hard to be a demonstrably badly performing private school - people leave and you close down. Not so with public schools where terrible performance can be hidden or sustained for much longer periods of time. So the Mexican data provides evidence that there is another story beyond average test scores and that is risk aversion. If you want to ensure your child does not end up in a dysfunctional school, then you have to go private even though you can statistically, on average, get just as good an education in public.

Today, there is another post shedding yet further light on the paradox. In, Charter High Schools Increase Earnings and Educational Achievement by Alex Tabarrok there is a look at alternate measures of success - graduation rates, education attainment, and future income. Based on the reported research, the charter schools increase achievement by 7-13 percentage points.

This suggests another avenue of thought. If public schools are losing the least well performing students at a greater rate than the charters, then you are no longer comparing apples-to-apples. So if the public school graduates 50% of its students and they are the best students, then if charters graduate 60% of their students, it means that they are retaining and graduating some of the lower scoring students. It should therefore be that their test scores ought to go down. If they are averaging the same scores but with a larger sample of the student performance curve, then that means they are actually teaching much better. This is an example where a reasonable performance measure, test scores, is an insufficient proxy of the real parental objectives and is masking superior performance.

If all you are concerned about are average scores, then it doesn't make a difference whether you send your child to a public school or a charter school. On the other hand, based on the data from Mexico, Florida and Chicago, if you have a balanced scorecard that is more strategic - i.e. test scores, graduation rates, risk aversion, education attainment and future earning capacity, then you absolutely choose the charter school..

What appeared to be a paradox when looking at only a single measurement, disappears when looking at a portfolio of measures reflecting a range of parental objectives and concerns. Highlight the importance of goal clarity and proxy measure selection.