Over the course of the years, we’ve happily written about scores (probably hundreds) of reports that are focused on broad areas of state and local government, like budgeting, human resources, performance management, infrastructure and so on. And we’ve been involved in creating some of these, too.
When we read one of these documents, we always try to see how the authors have reached their conclusions. We tend to be persuaded by quotes from experts who have strong reputations, clear-cut methodologies and most of all persuasive data.
Here’s the unfortunate news, that we hardly want to admit to ourselves. Despite the best intentions of many of these research exercises, there are often hidden flaws. Sometimes, these are just sloppy errors. Anyone can make a mistake.
But we’ve come to believe that all too frequently, the authors were fully aware of these issues, but didn’t want to share these dirty little secrets with the world.
Often, we only become aware of this situation when we get somebody on the phone to learn more about their research and discover that they can’t answer many of the basic questions that we pose, including concrete examples of the phenomenon about which they’ve written. We’re tenacious about such things, and generally we’ll ask our source to follow up and send us the missing facts or anecdotes, and in most cases they agree.
Then a day or two passes and we follow up with an e-mail asking for the missing numbers or the stories that will help us to illustrate their contentions. Sometimes the answers are forthcoming with a little nudging.
But often those e-mails go ignored until we contact the person a few more times and are ultimately told that they can’t get the information we’re seeking, either because they don’t have time or because the data aren’t available.
On our more misanthropic days, we conclude that they knew all along that there were no answers that they’d be able to find but were hoping that we’d forget and go away.
Ultimately, this leads us to using hedge words in our write-ups because we’re not entirely confident of the findings, and so we don’t want to hang our reputation on their absolute accuracy. Or in more extreme cases, we just drop the example from our own published work.
Here’s a real-world example: About six months ago, we were writing an article about a city survey. Our question was simple: “How many people responded?” There was no answer immediately forthcoming, but (using the approach we’ve just written about) we waited to find this out. if there weren’t enough responses, then we didn’t want to use the findings. Finally, our source told us that this information wasn’t public.
We could conceivably have used the Freedom of Information Act to dig deeper, but this was just one illustration we planned to use out of many. And we were on deadline. Our solution: We dropped mention of the survey from our piece.
But had we not thought to ask the original question, we would have run the risk of using the results of a survey that was problematic.
Another issue: Reports that make a strong point, include supportive evidence, but ignore a stockpile of facts that take the other side. We’re not suggesting that reports needs to say “on the one hand and on the other hand,” equating minimal evidence of one point of view with more powerful proof of the other.
But when there’s sufficient research showing a contrary point, we’d argue that the researcher owes it to the audience to acknowledge that.
All this brings us back to a time, many years ago, when we were writing a documentary about Walt Disney, for which we interviewed 77 people, most of whom knew him well, while others were well-known historians. In a filmed conversation with one of the historians, he told us that Disney’s father had never had any success in life.
We pointed out something we knew that was contrary to that, and the historian said the following: “Yes, that may be true, but it doesn’t fit into the theme.”
Years ago, an editor of ours (who we’ll not mention by name, for obvious reasons), complained that we hadn’t come to our conclusions before we did the reporting. We were somewhat younger then and didn’t have the courage to say that we thought this was ridiculous. We couldn’t conjure up any conclusions on the spot, which didn’t make him happy.
This kind of mindset was extreme, and we think it’s relatively rare. We believe most people try their hardest to be fair and honest and complete in their work. But too often that’s not the case.
And that’s a problem, particularly when other researchers rely on false published narratives, the false conclusions can be repeated until everyone believes they’re true.
#DedicatedToStateandLocalGovernment #StateandLocalGovernmentResearch #HiddenResearchFlaws #DataQuality #StateandLocalResearchFlaws #QuestionableSurveyResearch #ReportBias #MissingData #StateandLocalGovernmentManagement #StateandLocalGovernmentResearch #CitySurveyResearch #StateandLocalSurveyResearch #StateandLocalGovernmentReports #BandGReport #BarrettandGreeneInc
Comments