Posts Tagged ‘analysis’

The Value of Perspective

Jun 2010

20

102_5480 copy

 

Perspective is a funny thing. It’s so important to our everyday decisions in life, yet something that is commonly overlooked in analysis. Every decision we make is based on perspective. As data becomes more and more available, a large proponent of people who do not know how to take that data into perspective grows. They’ll overvalue certain sets of data without taking into account small sample size or confounding variables that lead to faulty conclusions.

 

The problem with the increased availability of data is that we start to focus SOLELY on the numbers, without thinking about the vast importance of perspective and relationships. Ultimately a number is just a number. Whether we’re looking at web traffic, batting averages in a baseball game or a company’s stock. Without perspective, we have no idea whether 1,000,000 pageviews is a good thing, a .250 batting average is a bad thing, or if a $20 stock valuation is the right price. It’s only once we take all factors into account (competition, peripheral factors, causal relationships), that we can truly begin to see the big picture.

 

Baseball: A Case Study

Nolan-Ryan-Baseball-CardIf you’re heavily interested or involved in analysis, then baseball is probably the sport you want to get into. There is no sport that has a longer, more rich history of statistical measurement and analysis than baseball.*
*For the record, I am extremely biased on this subject. Not only did I play baseball since I was 5 years old, but I also credit baseball for my knowledge of simple math skills and ability to do fractions (having to calculate batting averages and ERAs does wonders!)

 

So, since every statistic can be tracked, it allows people to easily (and often) draw unfounded conclusions based on certain data points. Someone might look at a batting average of .350 and think “Joe Smith is a fantastic hitter!” without taking into account certain other facts like: the pitchers he’s faced (maybe he only played in games where lower-tier pitchers were throwing), the number of at-bats (small sample size is often a huge factor), an unrealistic batting average on balls in play (if the league averages .300 when they put a ball into play, and Joe Smith averages .450, maybe he’s getting unreasonably lucky), etcetera.

 

Even if his average is legitimate, we must then decipher the reasons for his success. Maybe he’s found a new hitting coach, changed his batting stance, or changed his workout program over the summer. Maybe Joe Smith moved to a new baseball park that suits his style of hitting. Perspective when measuring and analyzing any statistical set of data is extremely important, and baseball statistics illustrate this perfectly.

 

Web Analysis

So, this brings us to web (and social media) analysis. Just because we have tools to measure all of our web traffic doesn’t mean that we are truly understanding what it is that we are measuring. Still, we often run into issues of targeting the wrong metrics and singularly focusing on data rather than contributing factors.

 

Many people typically focus on certain mainstream metrics (unique users, pageviews, time on site), without ever analyzing or paying attention to the peripherals (entrance page, referring sites, bounce rate). Focusing on a website’s peripherals allows us to realize not just that people are coming to a website, but more importantly, why.

 

The other big issue arises when choosing to form conclusions based on initial hypotheses. For example, thinking: “if there’s an increase in traffic, it’s because of our marketing efforts; if there’s a decrease in traffic, it’s because of a problem with our content.” As any 10th grade science teacher can tell you: it’s wrong to base your conclusion upon the hypothesis that you started with. Yet when doing analysis, we often tend to see what we want to see. Yet, just as in baseball, there are hundreds of factors involved. Seasonality (what time of year is it?), decreased demand for a product, or many other confounding factors (maybe someone else stumbled upon an article of yours) sometimes act as much stronger factors than your own efforts. You should always start with a hypothesis, but being unwilling to bend if that initial theory comes into question (or being uninterested in digging deeper) can often lead to incorrect conclusions.

 

Let’s take an example of a company that has been trying to ramp up social media efforts. They’ve started a Twitter and a Facebook account, and they suddenly see a great spike in their traffic. Quickly breaking out the Microsoft Excel spreadsheet, the social media specialist (who is tracking their web stats with Google Analytics) sends his CEO the chart below and says “hey, look! The main traffic dashboard shows a spike right when we launched a new social media campaign!” The company rejoices and finally sees the value of social media.

 

social media campaign stats The Value of Perspective

 

…Only, that’s not actually what happened. Yes, the social media specialist launched a campaign. Yes the traffic boosted. But if he’d taken 30 seconds longer to actually investigate, he’d have realized where the traffic was actually coming from. It wasn’t a bump in traffic from the Twitter or Facebook domains, or even in direct traffic, which he might have been able to attribute to one of the two. Nope, it was a bump in traffic from stumbleupon.com to a blog post he’d written 3 months ago. But he was so excited to affirm his belief that it was the Twitter and Facebook launch that week, he didn’t even feel the need to do research.

 

stumble upon stats The Value of Perspective

 

Don’t be that guy. If you are actually doing analysis on everything your company does online, you’ve already taken one major step. Don’t negate that by being careless. Take into account all of the variables before you rush to a conclusion. We’ve all been excited about the campaigns we run, and the efforts we’re taking to increase sales, traffic, conversions, whatever. But then take the next step in making sure you’re using the numbers right to attribute your successes to the right place.

 

The increased availability of data is a gift, but one that must always be wielded with caveats and perspective. Data is not inherently good or bad, but it is imperative that we view and analyze it with the proper perspective before moving forth with conclusions. However, if you don’t care about factual accuracy or moral integrity, you would be far from the first to lie with statistics.

 

All I’m saying is that we are currently in an era where we are inundated with statistics. Numbers, data points and rudimentary analysis are constantly thrown at us. If we just take it at face value, or decide to solely use it to fit our hypotheses, we’re only hurting ourselves.

 

Attention All Analysts

Jul 2009

10

To whom it may concern,

 

I have noticed that for a long time (but especially recently), you’ve been picking up dirty habits. Considering there is no good system for vetting you, I would like to address these two issues here. Additionally, I’ll give a couple prime examples of your large blunders as well as a current example.

 

extrapolating Attention All Analysts
(Source: xkcd)

 

Extrapolation is Dangerous
Ex. 1: The value of radio stations continued to rise in the 1990’s. Corporate behemoths like Clear Channel saw the growth and thought they could buy up all the stations and have control over the airwaves. Due to the rise of the internet and the changes in the market, radio is dying and Clear Channel’s creditors are trying to force the company into bankruptcy.
Ex. 2: In the late 1990’s, AOL was a media super giant. Its valuation was so high that it allowed it to purchase Time Warner in January of 2000, when the sky was the limit for AOL with subscriber growth through the roof (deal finally closed in January 2001). Since 2002, AOL’s subscriber base has not seen one quarter of growth. Time Warner has announced that AOL will now be split into a separate company by the end of 2009, fully breaking ties with AOL.
Ex. 3: Facebook and Twitter have each seen exponential growth. Facebook recently turned down a valuation of $8 billion, and Twitter was valued at $250 million in January (Facebook tried to purchase it for $500 million).

 

correlation and causation Attention All Analysts
(Source: Stephen R. Johnson)

 

Correlation does not imply Causation
Ex. 1: People who drink coffee have health problems. [No, people who drink coffee also often smoke and are workaholics. Those people do have health problems.]
Ex. 2: Eating breakfast is highly correlated with success for students. [No, people who miss breakfast are often absent or tardy students, or parents who make their child's breakfast are more involved in their lives (read: schoolwork, success)]
Ex. 3: As ad spending on a product increases, so do its sales.

 

Analysis doesn’t just take into account numbers and graphs with lines pointing in one direction or another. A good analyst will look at the other (sometimes hidden) variables and shed some light on whether the data shows mere correlation or actual causation. To that end, an analyst has to keep in mind that there are thousands of variables that will change over a one, two, five year period, which is what makes extrapolation so dangerous. Predicting Facebook’s pageviews in 2013 is no more certain than predicting AOL’s subscriber base or the radio industry’s growth. Just because a pattern exists doesn’t mean that it will last. Or that it isn’t just coincidence. Keep that in mind the next time you write a report.

 

Please. Friends don’t let friends extrapolate or imply causation. It’s just poor form.

 

Sincerely,
Jaremy

 

Links from @jaremy Tweetstream May 17-30 2009

May 2009

31

I missed last week’s update, so here is the past two weeks’ Tweetstream links.

 

CategorySiteDescription
Gaminghttp://bit.ly/bEenuTiger Woods PGA Tour iPhone review
Gaminghttp://bit.ly/p7BInMass Effect 2 teaser video
Gaminghttp://bit.ly/pjH3BinFAMOUS comic-style marketing campaign on IGN
Gaminghttp://bit.ly/17QPkWLionhead Studios to present at Microsoft's E3 presentation
Gaminghttp://bit.ly/13RcTFTeam Fortress 2: Meet the Spy
Gaminghttp://bit.ly/f7iIHDuke Nukem Forever's development team canned
Gaminghttp://bit.ly/envQQE3 Floor Plans
Gaminghttp://bit.ly/37ovmPopCap VP heads to Blizzard
Gaminghttp://bit.ly/1aw8sOTrailer for Call of Duty: Modern Warfare 2
Gaminghttp://bit.ly/XYi38Natural Selection 2 trailer
Gaminghttp://bit.ly/1aBrqqDell exits the gaming scene
Gaminghttp://bit.ly/W3nZNLeaked videos for Assassin's Creed 2, Splinter Cell: Conviction
Technologyhttp://bit.ly/11ybwvApple considering enabling background apps on iPhone
Technologyhttp://bit.ly/iJTNYYou can close apps in Windows 7 with Mouse3
Technologyhttp://bit.ly/tNvbnGoogle Wave preview - revolutionary or just cool?
Technologyhttp://appft.comThere's an app for that - very funny Apple parody website
Technologyhttp://bit.ly/4Ri5MGoogle automatic email message translation
Technologyhttp://bit.ly/18xIHdFans trying to bring back Polaroid film
Technologyhttp://bit.ly/169McCZune HD official announcement
Technologyhttp://bit.ly/Uvb8nMicrosoft Bing hands-on
Analysishttp://bit.ly/11BpELMaster your own social media data flow
Analysishttp://bit.ly/19AJOeRandy Johnson has won 249 games since turning THIRTY years old
Miscellaneoushttp://bit.ly/1PD46Danger Mouse's new album will be released entirely online (through torrents)
Miscellaneoushttp://bit.ly/XZtTnRunPee website tells you when to pee during movies
Miscellaneoushttp://bit.ly/1as8iHA mug for pessimists
Miscellaneoushttp://bit.ly/kOPpjWatch MST3000 24/7 at Justin.tv
Miscellaneoushttp://bit.ly/FpIxGFormer employees' recycled business cards

 

Technorati Profile

 

But who analyzes the analysts?

May 2009

25

sales But who analyzes the analysts?Every day there are dozens of analysts that predict huge sales numbers* or that make claims like Facebook Could Surpass Google by 2011. These predictions make it out to multiple media outlets, are blogged, tweeted and mentioned by hundreds of other outlets, and are discussed at great length. Additionally, there are thousands of analysts who make their living by making their own predictions.

 

The problem is that by the time the numbers actually come out, the predictions have been forgotten and the results rarely get reviewed.* So what’s to stop an analyst from making bold predictions when there are little to no repercussions? Aside from common sense, it doesn’t seem like very much.
*Full disclosure: I am an analyst. Predictions I’ve made are rarely openly vetted after the fact. Yes, I am aware of the irony of this rant.

 

I would love to see a website that takes analyst reports (whether the topic is gaming, social media, finance or sports) and actually vets the analysis. This is not to say that there aren’t analysts who do a good job of making conservative estimates based solely on facts. However, for the general public, it’s hard to differentiate between factual analysis and predictions based on a large number of suppositions.

 

To analysts: it’s time to stop sacrificing credibility for popularity. Anyone can make an outrageous assumption for the sake of appearing in headlines. That shouldn’t be the motivation. To news outlets and blogs: if you’re going to go through the process of printing analysis, you owe it to yourselves and to your readers to report on the results after the fact. If analysts are consistently wrong or off-base, their reports shouldn’t be published just for the sake of being newsworthy.