research bias

Research Bias: It Can Happen to Anyone

We are nearing the end of our year living in Europe. One of our goals for this trip was to experience the rich Roman and Medieval history in the area – most of which requires driving south. It is very quick to get around France once you are on the freeway, but freeway entrances are fairly far apart.

From our house, you can get on the freeway by driving north or south. The south entrance, we will call this Route A, is further away, but is in the right direction once you are on the freeway. Here is the Google Map from close to our house to the town of Orange.

The north freeway entrance, we will call this Route B, is closer, but is north from where we live so takes you out of the way. However, you spend more of the time driving freeway speeds. Here it is on Google Maps.

Now here’s the thing – my husband and I do NOT agree on which route to take, and driving in the car offers plenty of time to discuss why our particular point of view is the correct one.

My husband started with the stats from Google Maps to support his point of view:

Route A: 46.3 km, 47 minutes (shorter and faster)
Route B: 64.3 km, 51 minutes (longer and slower)

I pointed out that Google Maps is based on algorithms and not reality, so suggested we do some real-life measurements. The next few trips south I wrote down our departure times and arrival times at a set point on the freeway both coming and going (yes, we are that kind of couple!) using both routes:

Route A: 46 minutes, 58 minutes, 53 minutes, 52 minutes (faster 1 out of 4 times)
Route B: 50 minutes, 52 minutes, 51 minutes, 49 minutes (faster 3 out of 4 times)

Now, it should be immediately obvious that my husband is an advocate of Route A, and I prefer Route B. And we both have data to prove that we are right. My husband would point out that my measurements were invalid since on the trips where Route A was slower we ran into unusual traffic. I would point out that if we ran into unusual traffic 3 out of 4 times it probably means that the traffic is not that unusual.

Of course, we should know better. He is an engineer and I am a mathematician. Before we started all our “fact finding,” we should have just put on the table that we are biased. I don’t like roundabouts, so getting on the freeway sooner is a way to avoid them. My husband finds the freeway boring and prefers going through the towns.

So back to the market research topic. In this case the final result doesn’t really matter. But in any market research project, it’s important to know and confront biases at the beginning of a project, so you can ensure they are addressed. Findings must be strong enough that exceptions that support existing biases are identified and understood.

Market Research: Can you trust it?

by Diane Hagglund

Hal Varian, chief economist at Google recently said that the “the sexy job in the next 10 years will be statisticians”.

I think he’s right. Every participant in a first-world economy should have a solid understanding of stats. Not that I’m recommending that everyone become a statistician, but with the volume of information out there, it’s important to understand how data can be used to sway you.

Here are two “statistics” about my own life:

i) I was the only female graduate in Pure Mathematics at my university in my year (true).

ii) 50% of my graduating class who majored in Pure Mathematics were women (also true).

One of those statements paints a picture of a mathematics education that is oppressive to women, somehow subtly driving females away. The other paints a picture of a very progressive math department that gave women the same opportunities as men.

As I’m sure you’ve figured out, there were only two people in my graduating class that majored in Pure Mathematics. But without that information, I could have easily mislead you. Clearly the statistical significance is not there.

When someone presents information, whether a market researcher or anyone else, always think about the reliability. What’s the methodology? How big is the data set? Is the audience who completed the research the right audience to comment on the topic? Is it representative?

Research bias is very real. You need to be aware of this and always ask the right questions in order to determine whether information presented to you is valid – whether selling enterprise software or listening to the media.

Research Bias – How Different Participants Yield Different Insights

by Diane Hagglund

According to a recent survey from a specialist PlayStation 3 site, geeks make the best lovers.

Let me clarify right from the start that as someone with distinct geek-ish tendencies, who is married to a confirmed geek, I have no intentions of arguing with the study’s conclusions – just with its methodology.

The problem with the survey is that it was biased. Given the audience likely to peruse a PS3-specific site, it’s fair to assume the respondents skewed more toward the “geek” end of the spectrum.

When conducting research you must know who you’re talking to in order to interpret the conclusions correctly. If they had done the same survey with a running site, would they have found out that marathoners are the best lovers?

In technology market research, identifying your research target audience is especially important. Customers will give you different insights than prospects, employees, partners, or even the customers of direct competitors!

If you’re talking to your customers, you need to acknowledge that by definition they have the pain that you’re selling to, or they wouldn’t be using your product. So, you shouldn’t do a customer survey and then announce, “100% of the MARKET does this.” Of course, it’s completely fair to say, “100% of our CUSTOMERS do this.”

In comparison, if you’re talking to the general market who isn’t as fully educated on your solution, you’ll need to tailor your questions accordingly. If you ask questions that are too detailed and specific to your solution, you’ll get uninformed answers. The type of questions that are appropriate for customers who are familiar with your product, are not the same ones to use for the general market.

For example, if you’re asking about product features, you will get vastly different answers from existing customers who know your product, and from your competitor’s customers who know an alternative approach. Both of these will be different from the answers you’ll get from prospects who don’t have any product experiences.

As with everything in market research, it comes back to goals. Your very first step should be figuring out your business goals. Then, figure out what you need to know in order to achieve these goals. Next, figure out who has that knowledge, and who doesn’t. Only once you’ve answered all these questions, you can dive into the details of the market research project.