Dimensional Research

Technology Market Research

Market Research: Living With “Don’t Know”

by

We’ve read with great interest Alastair Gordon’s recent article on research methods measuring emotional response.

Mr. Gordon points out that sometimes, “Don’t know” may be the real answer after all. He adds, “Many people choose brands automatically, as a matter of habit and with little thought. Reasons for attachment can get buried and forgotten. Autopilot purchase behavior prevails in lots of categories.”

Mr. Gordon is referring to consumer products, but his argument applies to technology market research as well. While “autopilot purchases” are less common in Corporate IT than in other areas, they still happen.

We recently completed a study for a client that included talking to recent purchasers of their product. The topic of the study wasn’t competition, but there were some great competitive anecdotes that popped up, so we included them in the report. After all, who doesn’t love hearing about their competitive wins?

There was one particularly interesting scenario where the research participant already owned the competitor’s product and had done a successful implementation of the it. Still, they shelled out what was for them a significant amount of money to purchase our client’s product.

When this came up during the interview, I drilled down on the reasons for not using their existing solution – missing features, total cost of ownership (even though they product was free, maybe there were support or infrastructure costs?), relationship issues with the account manager – but it really boiled down to something else entirely.

There was no “logical” reason for the purchase. It was the power of the brand – although the research participant didn’t realize that and was unable to articulate it. He “didn’t know.”

Of course, this was very good news for my client, they have a great brand that wins them competitive deals. But the participant couldn’t articulate that. It was one of those cases where “Don’t know” was the real answer after all.

Can Research Do More Harm than Good?

by

First, an obvious point. We here at Dimensional Research truly believe in market research. We wouldn’t do it for a living or spend time blogging about it if we didn’t believe there was something fundamentally useful about going into the marketplace and getting feedback.

But a couple of things we saw and read this week made us pause and reflect on the value of what we do:

  1. A regular feature in a popular comedy show is “New Rules.” In a recent show, one of the new rules offered was “Online shopping sites must stop asking for my feedback.” This was the biggest applause line of the entire show. The audience loved it!

  2. We came across an interesting blog post from Tim Berry on Bad Research. Tim argues that it’s “Better to know what you don’t know than to make business decisions based on false information and false conclusions. If the focus group said red is better than green, nobody dares to argue for green. Even if green is really better, and the focus group was off, distorted by one very articulate and engaging green hater. Red it is.”

I understood the audience’s reaction to the “give me feedback” line. It’s frustrating to be asked to give feedback when you don’t believe it matters or don’t believe it will change anything. And I get where Tim’s angst about research comes from, especially the scenario he talks about which looks like a clear case of a “Fascinating Outlier“.

Does the research industry need their own version of the Hippocratic Oath where we swear to “first do no harm?”: We solemnly swear that we will not frustrate customers while getting their input and we will not use research to mis-inform.

We think that the learning here isn’t not to do research, it’s this: Do GOOD research.

The Curse of the Fascinating Outlier

by

You know this one, right? The market research project with a great recruit, good feedback, and actionable information. The segments you targeted were well represented, and a comprehensive view of the problem being studied was presented.

Then it happens. The curse strikes! The discussion during the report presentation focuses on just one participant: Some individual who was dynamic, funny, and perfectly represented a specific view of one of the study sponsors.

So why is that a curse? If the fascinating participant is an outlier who represents a view that is not in line with the majority of the study, then you have a problem. The opinion of one participant should never be over-weighted, no matter how articulate they are.

A couple of tips for how to handle this:

  • Make sure the client observes enough participants to balance out any single point of view
  • Look for cues that you have seen the “Fascinating Outlier”. Then, in the “post-research huddle” immediately put the input in perspective. If you’ve seen enough participants, strongly point out how unusual the participant’s point of view was. If it’s early in the study, set the stage for different input with a comment like “we’ll of course have to see what turns up during the rest of the study.”
  • Specifically call out the outlier’s input in the report. Have a section labeled “Outlier Input” (or “Off-the-Island Input” if your audience isn’t market research savvy) that includes an analysis of the persona, why they were different, and why their feedback was not in line with the rest of the study.

Energetic, passionate, articulate research participants are great, and outlier opinions are important to balance any study. But following these tips will help put their feedback clearly in context.

Phone Surveys or In-Depth Interviews: What’s the Difference?

by

A while back we blogged about phone surveys, and argued that they do not have a place in technology market research. Several comments were made both in the blog comments and to us directly that made us realize that there is confusion about the difference between phone surveys and in-depth interviews (also called IDIs). Here’s how we view this:

Phone surveys and in-depth interviews are similar because:

  • Both are conducted over the phone.

But that’s pretty much the only similarity.

The differences between phone surveys and in-depth interviews:

Phone surveys: A quantitative research method that includes a large numbers of participants.

  • Capture input to a common set of questions with pre-set answer options
  • Administered by a phone survey professional that has a pleasant voice and is trained not to guide the responses of the participants in any way
  • Are usually short: 5-20 minutes as rule of thumb
  • Usually no incentive is given to participants, although there may be a small amount given.

In-Depth-Interviews (IDIs):A qualitative research method that uses a smaller number of highly select participants.

  • Screener is written to allow significant discovery, open-ended questions, and drill-down
  • Administered by a trained moderator who is versed in the client’s business, the goals of the research, the topic of study (in our case technology) as well as techniques for putting the client at ease and getting the kind of feedback desired
  • Usually longer: 30 minutes to 1 hour is common
  • Typically generous incentives are given to participants to compensate them for the time commitment

An easy way to think of the differences are that phone surveys are pretty much like web surveys, but conducted over the phone. Given these definitions, we stick by our earlier recommendation that phone surveys have no place in technology market research. They are significantly more expensive to conduct than web surveys without adding value in a corporate IT study (although agree that there are audiences phone surveys are appropriate for).

That said, there is nothing inherently wrong with phone-based research. In fact, in-depth telephone interviews have become one of our most valuable research methodologies, particularly as clients become more global and travel costs are becoming more of a factor in evaluating research budgets.

5 Questions To Ask Before Engaging A Market Research Firm

by

I went back and forth about writing this post in case it seemed self-serving, but it’s a question that we get a lot, so I wanted to offer some guidelines to consider when choosing a vendor for a market research project.

1) What is the firm’s expertise?
Does the firm you are dealing with know anything about your target market? At Dimensional Research, we work exclusively with technology. Our researchers have spent their entire careers working with the people, processes, and technology used by Corporate IT, and I can say very confidently that we know our stuff.

This matters for two reasons. If it’s a technology-related research project: (1) We’ll get better results than any research firm that doesn’t know technology, because we’ll understand the language being used and the content it’s being used in.  This means we do better follow up and analysis. (2) We won’t consume a lot of your team’s time with “hand-holding” while the researcher comes up to speed. We’ll dive right in and be productive immediately.

2) What is the researcher’s expertise?
Once you have found a firm that is a match, make sure you talk to the person who will be doing the actual research – conducting the interviews, moderating the focus groups, writing and analyzing the Web survey. Is that person an expert also? Not every researcher at a firm knows everything the firm has experience with. Ensure you have the right person or team of people doing the hands-on work or the experience the firm brings will be essentially irrelevant.

3) Does the firm demonstrate expertise in their own marketing?
So you think you’ve found a good market research firm? Now look critically at their marketing. This will depend to some degree on the project of course, so look at the appropriate items:

  • If you have someone doing Web usability research, is their own Web site design usable?
  • If you’re doing a message validation, how good is their messaging? Is it compelling? Did they understand your pain?
  • If they’re doing competitive research, what do they know about their own competition and can they articulate a differentiator?
  • If it’s a customer satisfaction project, how satisfied are their own customers? Do they track customer satisfaction? How?

4) Has the firm/researcher done a similar project before?
If the firm’s answer to this question is “yes,” get a reference call set up, or a very, very good reason why not to set up such a call. Researchers tend to develop very close relationships with their clients – during a project there is a lot of communication – and if you can’t get a reference, that’s a big red flag.

If the answer is “no,” that’s not necessarily a deal breaker – there is a first time for everything – but is your research firm honest about that? Are they willing to be open throughout the process so mistakes are caught early?

I’ll make a confession – the first time Dimensional Research tried using social media to recruit it didn’t go very well. We were still learning the ropes and didn’t understand where our audience was hanging out. But the client knew we were trying something new, and we had an extra two weeks built into the schedule to course correct. Good thing, because we needed it, but the end result was a very successful project.

5) THE MOST IMPORTANT ONE:  How have they evolved their own business based on their research?
I was shocked to meet a fellow researcher recently who essentially hasn’t changed their business in ten years.  If a research firm doesn’t fundamentally understand the value of research enough to use it to make their own business better, DO NOT hire them.  A good researcher always has their research hat on for their own business.  Dimensional Research treats every engagement as ongoing research into our own business to launch new products like our ROI analysis.  We also constantly test our assumptions, as in this example of what we did with stipends.

The other great thing about question #5, is that it can be answered by any firm.  One of the frustrations of market research is that the vast majority of our work is confidential, so it’s impossible to share our biggest successes.  But any firm should be able to articulate how they use research to drive their own business.

Should Market Research Be Fun for Participants?

by

I was reading an interesting market research article by Tim McAtee. This line near the end of the article really jumped out at me: “…this approach can make research more enjoyable for the subject by making survey vehicles less boring.”

This raised a couple of questions for me:

  1. How much fun do our participants have when they are engaged in a project with us?
  2. Does it matter to the outcome of the research?

Fun can certainly add value. Any interviewer knows that getting someone to relax during the interview results in more honest answers and better research findings. We purposely write our interview guides to start off with more general topics so we can get the participant engaged.

Of course, you do have to cover the topic of the research well, but if the participant is qualified to talk about a topic, they probably really enjoy talking about it. And of course a completely boring survey, especially with too many ranking or rating matrix questions, will have a huge drop-off, or if the stipend is really good, they’ll speed through the survey to get the completes without giving thoughtful answers.

On the flip side you can have some bad results in a focus group when there is too much positive energy floating around the room – which is just as bad as an overly negative atmosphere. You can’t have an environment where people are hesitant to share negative viewpoints because they don’t want to bring everyone down.

IMHO there are some definitive statements about fun and market research:

  • Boring is NEVER good for market research.
  • Participants who enjoy the process  give better feedback.
  • Fun cannot come at the expense of research goals.

I have one more theory: If the researcher is engaged and enjoying the study, that helps the outcome too. Here at Dimensional Research, we love what we do! We find the IT professionals we have the pleasure to work with endlessly fascinating. They have a truly difficult job and their creativity, tenacity, and smarts consistently impress us.

Three Signs That You Are NOT Ready For A Web Survey

by

We hope that we’ve made it clear in this blog that Dimensional  Research is a big fan of Web surveys.   Web surveys are a great market research tool. They make it easy to get immediate results right at your fingertips.

However, you need to have a certain level of knowledge before you can run an effective Web survey.  Here are three signs that might indicate you are not ready for a Web survey, and you should do some qualitative research first.

  1. If you write multi-choice questions and are completely guessing the answers, you’re probably not ready for a Web survey.  Of course, you always need to have an “Other” section to cover the corner-cases that you just didn’t think of, but your options must capture most of the likely responses to effectively quantify a finding.  Let’s admit it, survey takers will often pick a a presented option that isn’t quite right rather than take the time to fill in an open-ended option, so you can’t rely on “other” to cover your lack of knowledge.
  2. If you’re asking a lot of open-ended questions, you’re probably not ready for a Web survey.  But what is “a lot?”  Good rule of thumb, no more than one open-ended question for every 20 survey questions – not including that important last question “Is there anything else you’d like to tell us?” that you put at the end of every survey.
  3. If you get only one shot at an audience, don’t waste it with an uninformed Web survey.  If you don’t completely understand a new market, and have a participant list that you can use only once, mitigate the risk that you’ve gotten something wrong by doing a few interviews with a couple of list members, or doing some small trial surveys with more open-ended questions that you can use to form a better Web survey.

Don’t waste a Web survey. If participants are taking the time to give you feedback, make sure you’re getting the most from them.  Don’t be scared to add some test surveys or interviews to a project schedule – the extra week or two needed in the schedule will give dramatically higher results.

We just wrapped up a great project that started as a stand-alone Web survey. We needed to quantify some specific purchasing metrics, so the survey was clearly the right methodology.   But as we drilled deeper, we realized that we were making far too many guesses about actual purchase motivations.

We added a series of 15-minute customer interviews prior to the Web survey that gave us some great insights into the  scenarios that were driving behavior, and then developed a much better Web survey that gave crystal clear data about all scenarios. Most importantly, it allowed us to eliminate a group of customers that weren’t motivated by the conditions we were evaluating (although their behavior was similar) and would have skewed our data significantly if we hadn’t excluded their responses for certain questions.

Going into the final presentation, we were very glad that we’d done the interviews.  We knew our stuff cold and had the backup we needed to defend the results and ensure they were taking seriously – a must to influence the business outcome.

Corporate IT Adopting Windows 7: New Research from Dimensional and Dell KACE

by

Dimensional Research has just completed the fourth in a series of surveys sponsored by Dell KACE.  Over 900 IT professionals completed this survey on the adoption of Windows 7.

The news is very good for Windows 7.  More than half plan to deploy before the end of the year with many IT departments not planning to wait for SP1 to deploy.  Concerns about performance and reliability have dropped significantly since the release of Windows 7.  Key findings include:

  • 87 percent of survey respondents plan to deploy Windows 7 compared to 47 percent who had plans to deploy Vista at a comparable point after its release;
  • 46 percent of the total surveyed revealed they have plans to migrate even before the release of SP1;
  • 86 percent reported concern about software compatibility when migrating to Windows 7;
  • 25 percent expressed concerns about Windows 7 performance, down from 47 percent reported during the 2009 survey;

The study reveals an interesting opportunity here for products like those offered by Dell KACE since 72% of participants plan to do their Windows 7 migration manually or using free or point imaging tools.

The press has been covering this report, with some of my favorite stories here:

You can download a copy of the full report here.

Rethinking Market Research Incentives

by

Incentives are a normal part of any market research project. With each project we ask ourselves: What will we give participants?  How much will it cost?  How do we sweeten the pot to ensure high participation?

Today we’d like to take a step back and ask:  Do we really need to give people stuff to make them talk to us?

Now, before anybody panics, we do believe that most of the time the answer is “yes.” When we’re reaching out to new audiences that we have absolutely no relationship with, we need something to get their attention.  And if we’re asking for something big – driving across town to attend a focus group for example – of course the stipend is important.  We’ve discussed guidelines for compensating participants in the past, and we still think that post is relevant and useful.

But that doesn’t mean that we always need a “prize.”  Especially when working with customers, the best reward might be that they see the feedback they give influencing the product roadmap or support. Sometimes just being heard is a very strong driver.

For example, we have just completed an annual customer survey for a customer we’ve been working with for years. The first 3 years we gave a typical stipend of a $25 gift card to anyone who completed the survey.   But, this year the budget was slashed, so we figured we’d try it without a stipend and see what happened.  A couple of observations on this test:

  • We got the same number of completes as last year – when we did offer a stipend!  The customer base had grown by 15% so we can expect that the lack of stipend hit our response rate by that much, but we still had a very significant response rate so the results were valid.
  • Responses were higher quality than in the past (as evaluated by the thoughtfulness of responses to open-ended questions).  Since nobody completed the survey just for the $25, participants were more engaged.  They took the time because they cared.
  • Perhaps this wouldn’t have worked the first time? This was the fourth in a series of customer surveys, and this client has definitely responded to the findings of past surveys.  Perhaps their customers now realize the survey is worth their time for other reasons then the stipend?

Net-net: We felt we got at least as good as – and arguably better – results from the customer survey without stipends for less than half the costs of previous surveys. Moving forward, Dimensional Research is making sure we evaluate whether the stipend is truly adding value to a project.

Market Research and “Ethical Treatment of Human Subjects”

by

I don’t know the AAPRO well enough to know if they were trying to be funny, but I laughed at the language they used to write this survey particular tip:  Maximize cooperation or response rates within the limits of ethical treatment of human subjects.

(Thanks to Cathy Harrison and  Bernie Malinoff for Tweeting this article.)

Medical and psychological researchers have strict ethical guidelines in place dealing with people in their research, maybe people who run Web surveys need some also.  Here is our input into guidelines that should be included in a manifesto for being kind to human participants in market research studies.

  1. Never force someone to give you a wrong answer – When Steve Carrell gave Conan O’Brien his exit interview on the Tonight Show and asked if working at NBC was “great”, “really great” or “fantastic”, I had a flashback to real-life employee survey I had to take years ago.  Web surveys should always cover every possible option someone could give or offer an “other” option.  A good survey will have a comments area at the end to allow users to clarify anything they felt didn’t represent their input.
  2. Use complex ranking or rating matrices with caution – It is just plain painful to answer long rating or ranking questions even when you deeply care about the subject. You always end up missing at least one line and get that oh-so-not-helpful “please answer number 14”.  Be kind and use matrices sparingly and only when there are no other approaches to get the answer you’re looking for.
  3. Don’t ask questions that simply don’t matter – Seriously, does my ISP need to know if I’m a man or a woman?  Is that going to change the way that they deliver service to me?  If it does, then by all means ask, but I frequently feel like somebody took a market research 101 class back in their first year of college and ever since have opened with demographic questions on age, gender, and ethnicity independent of if it matters to the survey goals.  There is something to be said for asking “warm-up” questions that are easy to answer, but surely there is something more interesting then gender to ask.  [UPDATE:  Of course there is always the “Elinor Exception” to this rule as rightly pointed out in the comments.  If you need to add a sanity check on the quality of the sample, then go ahead and ask the gender question.]
  4. Don’t expect participants to know all your lingo – Seriously.  Speak plain English  (or French or Spanish or Chinese or Japanese or whatever language your participants do).  Don’t throw in tons of jargon without defining it first.
  5. Be realistic about how long your research takes – It seems as if every survey in the world is “10 minutes” long.  Surely you can have at least one person who didn’t write the survey questions try it out and tell you how long it realistically takes an average person.

I’m sure at Dimensional Research we have on occasion broken one of these rules, but we try our hardest to ensure our participants don’t suffer in the process of giving their feedback.  We’re always happiest if they actually enjoy it.

What would you add to a “Manifesto on Ethical Treatment of Survey Takers”?