I asked Morse how U.S. News’ methodology differs from those used by other publications like Newsweek and the Washington Post. Here’s what he told me:

Their rankings are based on one factor – the number of (Advanced Placement and International Baccalaureate) exams taken compared to the number of eligible students. That doesn’t factor in pass rates; it’s only based on participation. Our rankings consider statewide assessments, and we ask people to understand the concept of relative performance. It’s not as simple as saying “the best school always has the highest results.” We’re saying on a relative basis a particular school is doing better than expected given the mitigating factors.

In other words, it’s actually possible to have a strategy of having kids take the AP and IB exams in massive numbers and get on the other rankings. In ours, the students actually have to learn something.

To be fair, just because a student doesn’t earn a passing score on one of the advanced exams doesn’t mean they didn’t learn something. In fact, there’s research indicating that students get long-term benefits from access to the more challenging curriculum of AP classes even if they never attempt the exam, including better outcomes when they move on to college.

The Washington Post’s rankings tally the nation’s “most challenging” high schools, which is a different benchmark than the one set by U.S. News. Jay Mathews, the Washington Post’s education columnist, helped to devise the so-called “Challenge Index,” and he’s been asked so many times to defend the methodology that he’s put together a helpful response to the most frequently asked questions. Here’s what he said about the decision to give schools credit for the total number of students who take AP and IB exams, rather than just counting the ones who pass:

To send a student off to college without having had an AP, IB or AICE course and test is like insisting that a child learn to ride a bike without ever taking off the training wheels. It is dumb, and in my view a form of educational malpractice. But many American high schools still do it. I don’t think such schools should be rewarded because they have artificially high AP or IB passing rates achieved by making certain just their best students take the tests.

As for the usefulness of these kinds of comparisons, that depends heavily on the circumstances. Like most education reporters, over the years I’ve fielded many phone calls from parents contemplating a move who want to know the “best” schools before settling on a neighborhood. The reality is that what works for one student — like the magnet schools, specialized programs, and charter schools that crowd the top spots in the U.S. News rankings — might not work for another. In many school districts enrollment is set by geographic boundaries and there are few opportunities for a family to request transfers to higher-rated campuses. All they can do is unpack the moving van, and hope for the best.

Have a question, comment or concern for the Educated Reporter? Email EWA public editor Emily Richmond.