WHY MANY STATE AND CITY RANKINGS DEFY REALITY

Back some years ago, when we first started to evaluate management capacity in states, counties, and cities for the now defunct Financial World magazine we were forced by the editors and publisher to rank the entities we were evaluating from best to worst.

We hated that for many reasons. As far as we could see the difference between number 29 and number 30 wasn’t even marginally significant and yet these comparisons were often picked up by the local press. That made the publisher happy as he loved to get lots of attention, but it never ceased to bother us.

Subsequently, when we began our work on the Government Performance Project, we took great care to make it clear that while we were evaluating and even grading the states, we weren’t ranking them. We carefully avoided ever using that word preferring to refer to our “evaluations”. Perhaps the GPP, which utilized the skills of many highly regarded academics and a team of journalists didn’t stir up the same kind of media frenzy than the far-far-far less-rigorous Financial World work (which was entirely done by the two of us), but the leadership at Pew and Governing were more interested in contributing solid useful information to the world of public sector management than they were in creating a stir.

In the years that have passed, it seems to us that there must be some kind of gold

mine in the field of publishing 1-50 rankings of the states and similar lists of best and worst cities. And we cringe when we see many of them, for a variety of reasons.

Forbes (which seems addicted to these kinds of lists) went so far about a year ago as to publish a 50-state ranking titled “States With The Most Devoted Dog Owners.”  According to the article, the ranking was based on a survey of 10,000 dog owners (200 per state) and compared them across seven metrics, including “the percentage of dog owners who broke up with a significant other who didn’t like their dog.” Apparently, “6.78% of dog owners broke up with a significant other who didn’t like their dog.” Woof.

Beyond the dubious nature of this kind of metric, and the value of such a list in the first place, the idea that you can get a solid sampling by asking 200 people from every state, regardless of its size, has zero merit. California dog owners were represented by about .0005% of its population.

We don’t want to get distracted by criticizing this kind of foolishness, though. That’s like shooting fish in a barrel. We’re far more concerned about rankings that are taken somewhat more seriously. For example, though we won’t be the first or the last to complain about the value of the U.S. News rankings of universities, they’re worth mentioning here.

For starters, these rankings always seem like a dangerous exercise to us, as we see families making decisions about college selections based on these rankings instead of the value of the program to which the high school senior is applying.

Beyond that, there have been plenty of criticisms of the methodology used to make these lists. Beyond the specifics, there have been lots of complaints about the ever-shifting methodology which makes for significant changes in the rankings themselves. As Daniel Diermeier, chancellor of Vanderbilt University wrote in Inside Higher Ed,  “Does this mean those of us who’ve fallen in the rankings are objectively worse than we were a year ago? Does it mean a university that shot up the list is suddenly orders of magnitude better? Of course not. The shifts in rankings are largely due to the changes in methodology.”

This raises two questions: Was the last year’s methodology wrong and that’s why there was a change? Or is it in the interests of the publication to see changes from year to year in order to make the horse race more exciting?

If all this wasn’t cause for concern about the validity of these rankings then consider the  January New York Times article that pointed out that “U.S. News sells ‘badges’ to colleges so they can promote their rankings – whether they are 1st, 10th or much much lower.”

While the college rankings are probably the best known, there are also a plethora of lists of “best places to work.” We can’t begin to enumerate all the potential flaws in these lists, but the degree to which they vary from ranking to ranking isn’t a very good signal that they should be regarded as valid.

For example, one list of “the best and worst states for work-life balance,” indicated that New Hampshire was the best of the lot. But then there was another ranking that claimed to demonstrate that New Hampshire was the worst state to be a teacher. Don’t teachers care about work-life balance? We’ll bet they do. Let’s say for the sake of argument – and we don’t believe a word of it – that both lists were accurate, teachers reading the first one could be heading as fast as they can to New Hampshire only to find out that in their profession they’d be better off anyplace else.

Finally, let’s think a bit about the “best places to live list.” Best for whom? These are almost always blunt instruments for coming up with a very complicated answer. Some lists use the level of home ownership as a measure of a good place to live. But that would mean that Manhattan is probably not the place for you, where high costs mean that only about 24 percent of the population own their own place. But there are clearly other reasons some people love living in Manhattan. We did for over 35 years and cherished every minute of it. All while paying rent.

One more: Let’s say that in your opinion low taxes are a wonderful way to pick your home state. Lots of lists rank the states by that criteria and you’d be led to believe you should head for Florida, which is famous for its exceedingly low tax burden. But d0 you have children in schools? Then it may be important that your teachers are well paid, and on that measure, Florida could hardly do any worse. Take things a step further and assume that you only care about low taxes and have no interest in the children of the state – but you happen to be a member of the LGBTQ community – well we don’t need to say any more about that.

#StateandLocalManagement #StateandLocalManagementRanking #FlawedStateRanking #FlawedCityRanking #FlawedBestPlacesToLiveRanking #FlawedUniversityRanking #USNews&WorldReportCollegeRanking #ForbesRanking #GovernmentPerformanceProject #FinancialWorldStateRanking #FinancialWolrdCityRanking #FinancialWorldGovernmentRanking #GoverningGradingtheStates #InsideHigherEducation #SillyStateComparisons #StateRanking #CityRanking #BestPlacestoLiveRanking #CollegeRanking #B&GReport