Anecdotes and Data

by Krishna on August 6, 2009

Joshua Porter had a good post on over-relying on anecdotes:

We take shortcuts, we tell anecdotes, instead of looking at the data. In the design world this happens all the time. People extrapolate from their own experience or stories they’ve heard about using the web, generalize it, and share it with others. That’s how anecdotes happen. But very rarely can we make concrete decisions based on them.
So the next time someone tells a wonderful story that sounds a tad too good to be true, ask yourself: “is this an anecdote, or is it real data?”.

We take shortcuts, we tell anecdotes, instead of looking at the data. In the design world this happens all the time. People extrapolate from their own experience or stories they’ve heard about using the web, generalize it, and share it with others. That’s how anecdotes happen. But very rarely can we make concrete decisions based on them.

So the next time someone tells a wonderful story that sounds a tad too good to be true, ask yourself: “is this an anecdote, or is it real data?”.

And then he goes ahead in another post to use an anecdote about Apple:

I’ve heard it said again and again, but I had never seen an actual quote in which Steve Jobs says that Apple doesn’t do market research. I finally found one. […] Jonathan Ive in a recent interview (also worth reading) explaining how Apple’s primary goal is not to make money, but to make great products. […]

It seems to me that Apple has a pretty clear story here: “Make the very best products. Business will follow.”

I don’t mean to pick on Joshua; in fact, I am guilty of such inconsistency in the past myself. But this illustrates a big problem with anecdotes – we are most likely to believe them if they match with our beliefs. If we like something or somebody, we are likely to believe the good stories about them and be very skeptical of the bad stories. And the opposite if we dislike them.

There is a good reason behind this. Our original beliefs are based on a series of good, original impressions about the person or thing. And any good news reinforces that, while bad news seems inconsistent with what we already “know”. This is especially true of deep-rooted beliefs in emotional subjects like politics and religion. But at least those beliefs are understandable. Religious beliefs are with us from childhood and reinforced by concepts of divine reward and punishment. Political beliefs are based on philosophical differences and even when there is evidence for or against a viewpoint, sometimes it is not very conclusive. Extreme viewpoints are easily disproved, but even without them, we have a wide spectrum of religious and political ideas.

Doing the same thing with respect to business and technology is self-defeating. There is enough data to form your impressions about a company or product without having to drink the Kool-Aid from the people running the company. The basic problem is that when explaining their success, people are not so good at telling the truth. They like to say what they think people like to hear. Or they just plain don’t remember.

Most entrepreneurs I have seen and read about are, in a way, megalomaniac (in a good way!). They believe very strongly in themselves, their company and their products. This may or may not be in odds with what the public thinks of them. For every product created by Apple or Google, there is a competitor making a product thinking they have created the best thing ever. In fact, soon when Apple and Google directly compete against each other (iPhone/Android), each side will imagine the same thing.

In essence, asking people about themselves gives you misleading answers. And generally, they sound too good to be true. I have linked to this before, but here’s what Bob Sutton has to say on it (emphasis mine):

And it has always bothered me that once his “chimps” (as he [Jim Collins] called his research team) identified the good and great companies, they went back and interviewed people about the differences. It is sort of like asking players of the team that won the world series or super bowl (and reporters) what the secret to their success was after the series is over, and then comparing their answers to the players on the fourth place team. Compelling research by Berkeley’s Barry Staw and others shows that winners and losers offer much different recollections to explain what happen and why, regardless of the facts.  In particular, winners report having better leaders, being more persistent, more focused, and more cohesive — even in experiments where there is no real difference between groups, and the winners and losers were simply misled about their performance by some seemingly legitimate person.  So I don’t know who will win the Superbowl this year, but I can predict what the winning players will say, and so can you.

Comments on this entry are closed.

Previous post:

Next post: