Blog: The Educated Reporter

Rankings fail?

If you know me, you know I am inherently skeptical about ratings and rankings. As an authority of both cake and pie, I lowered my guard and developed high hopes for the Jezebel dessert March Madness, and look what happened: cheesecake won! As a PIE! Which is it not.

Daveen Kurutz McLaughlin of the Pittsburgh Tribune-Review called my attention last week to an effort from and Forbes magazine to identify the “best cities to live and learn,” or “the best schools for your housing buck.” They calculated the list using a combination of test data and home prices. It seemed odd to Daveen because the Pittsburgh Public Schools were featured, even though they are hurting academically and, as she put it, “barely made AYP this year.” We looked into the methodology.

According to GreatSchools, the “education quality score” was determined by state test scores and NAEP data. I asked an official for more detail, and she said that proficiency rates on state tests were moderated by a multiplier depending on how well that state did on NAEP—because, as we all know, state tests are not comparable. So a district with a high pass rate in California is indexed lower than a district with that same pass rate in Connecticut.

This is sort of mathy … but not mathy enough. It feels like an imprecise manipulation of data to me. You can do lots of fun stuff with statistics, and I get the gist of what they were attempting, but I just do not think you cannot use state NAEP data to draw conclusions about local districts (except when you are specifically addressing those that participate in the urban NAEP—which Pittsburgh, among many others, does not).

As well, Daveen realized that the median home price listed was just for the city of Pittsburgh, while the academic index was derived from test results at all schools with a Pittsburgh mailing address, which includes wealthy suburbs very much NOT a part of Pittsburgh Public Schools. Given that the cities were recommended because of a combination of home value and high test scores, this discrepancy renders the equation meaningless.

My radar was set off by the inclusion of Honolulu. There are many reasons to live in Hawaii, but given that they have such a bad budget crisis that kids do not go to school most Fridays, I would not be trying to send people there.