«
»

But who analyzes the analysts?

May 2009

25

sales But who analyzes the analysts?Every day there are dozens of analysts that predict huge sales numbers* or that make claims like Facebook Could Surpass Google by 2011. These predictions make it out to multiple media outlets, are blogged, tweeted and mentioned by hundreds of other outlets, and are discussed at great length. Additionally, there are thousands of analysts who make their living by making their own predictions.

 

The problem is that by the time the numbers actually come out, the predictions have been forgotten and the results rarely get reviewed.* So what’s to stop an analyst from making bold predictions when there are little to no repercussions? Aside from common sense, it doesn’t seem like very much.
*Full disclosure: I am an analyst. Predictions I’ve made are rarely openly vetted after the fact. Yes, I am aware of the irony of this rant.

 

I would love to see a website that takes analyst reports (whether the topic is gaming, social media, finance or sports) and actually vets the analysis. This is not to say that there aren’t analysts who do a good job of making conservative estimates based solely on facts. However, for the general public, it’s hard to differentiate between factual analysis and predictions based on a large number of suppositions.

 

To analysts: it’s time to stop sacrificing credibility for popularity. Anyone can make an outrageous assumption for the sake of appearing in headlines. That shouldn’t be the motivation. To news outlets and blogs: if you’re going to go through the process of printing analysis, you owe it to yourselves and to your readers to report on the results after the fact. If analysts are consistently wrong or off-base, their reports shouldn’t be published just for the sake of being newsworthy.

 

  • Ah - the hockey stick problem!

    When I cover market sizing in my classes I cover secondary research as a way to get some information inexpensively. But I warn about issues with over-inflated forecasts and definitions that don't match the entrepreneur's view for their new product.

    The problem is particularly acute for technology related products, I suspect because it is harder to hold the analysts' feet to the fire, especially when there are examples of exponential growth. At least some of the data is based on primary research (Instat for example), but I doubt that they do much scaling back of responses based on being conservative (e.g. 60% of Very Likely, 25% of Likely). (I don't know this for a fact). And then there's the reports that come from analysts talking to industry insiders; there is very little incentive to be realistic - except perhaps for the next reporting cycle. Smart entrepreneurs and investors ignore or scale back projections for these reasons.

    Good luck trying to convince people to be more realistic. And blogging exacerbates the problem. Professional reporters and editors are not very good at reporting accurately even if the data was OK. Add the blogger who has even less understanding of statistics and the problem is magnified. Joel Best's books on Lies and Statistics do a good job of explaining the issues of reporting on results.

    Mike Pritchard (That Research Guy)
  • Agreed. The incentives just aren't there. It's just too bad, as especially in blogs, opinion, analysis, prediction and fact are often intertwined far too often with little caveat or preface. Outrageous predictions based on statistical lies and extrapolation sell reports, which often seems to justify the means in an analyst's mind. It's up to the analysts to be honest and up to reporters to be wary of misleading their readership.

    That said, excuse me for my lack of optimism on the subject.
blog comments powered by Disqus