Dimensional Research

Technology Market Research

Qualitative Results without the Hassle of Qualitative Research? Really?

by

Frankly, we’re not convinced.  I’ve seen a bunch of headlines like New Technology Achieves The Same Results As Focus Groups In A Fraction Of The Time And Cost, and have even had a comment left on the Dimensional Research blog from a vendor who offers similar solutions.

This strikes us as so much nonsense. It’s not that these couldn’t be great products – we’re quite sure that these tools can be very useful, and give valuable insights when used to answer the right types of questions.  Our beef is the claim that they can produce the same results as focus groups.  To assert that any tool can replace qualitative research seems to demonstrate a clear naiveté about the power of having a well-moderated conversation. One of the things we’re quite sure these tools cannot do is ask a really great follow-up question.

Qualitative research – whether focus groups or in-depth interviews – is not about listening to a group chatter about whatever they want, or giving them a topic then letting them have at it. It’s about really listening to participants and asking the question that gets them thinking. Often it’s about guiding the conversation to get at those insights that are just below the surface. The best researchers have a knack for understanding what is NOT being said and finding out why.

What do you think? Have we missed the point of these new tools? As usual, I’ll point out that our business is technology market research – we work with vendors who sell to corporate IT, which is primarily B2B. Perhaps these tools offer more value in consumer research. Either way, we welcome insights from anyone who has gotten good results from a solution like this. If I’m an old fuddy-duddy (and perhaps my use of the phrase fuddy-duddy is proof that I am one), go ahead and tell me.

Or possibly the problem is that these tool providers have a messaging problem?  I’m assuming they have NO plans to sell their product through traditional market research providers if their press releases are so dismissive of the work that we do. Perhaps they needed to do some qualitative research on their messaging with some of us?  😉

A Researcher’s Confession

by

I admit it.  Although web surveys are one of our most popular research offerings, I strongly prefer qualitative research (focus groups, in-depth interviews).

I can’t help looking at the findings in web surveys and feel like I want to dig in and ask more questions.  The participants say things that don’t make sense to me and I want to know why.  Or they make short comments responding to open-ended questions that leave me with a dozen follow-on questions I don’t have the opportunity to ask.

When I present qualitative findings, I know I can answer any question that comes up with complete confidence.  With quantitative findings, I always know that there will be questions where the answer is, “We can’t draw any conclusions based on this survey.”

This came up again last earlier this month:  We just did a series of in-depth interviews where we wanted to understand perceptions about the cost of various alternatives.  All the participants in this study had identified themselves as product decision makers who had full visibility into costs – a requirement for the study.   If you had just looked at the first level of answers people gave, you would have thought that one of the tools we were looking at was very inexpensive compared to other options.

But because this was an in-depth interview, I got to ask that all-important “why” question. I quickly realized that while all the participants had been educated on the “line 3” costs that were billed directly to their organization, not everyone was aware of the additional “line 10” costs that had to be added to support this different approach.  When you added both of those up, the tool that originally appeared less expensive turned out to have a similar TCO to other options.

Now, it’s true that we could have found this out by writing a good web survey, but one of the secrets to writing great web surveys is to know the answers to all the questions first.  We continue to recommend web surveys as good vehicles for quantifying concepts that you know well, but want to put an accurate % by each of the options.  This is a valuable thing to do, especially for market sizing, external marketing and PR purpose.

But for finding out the answers that you don’t really know, start with qualitative research – and by all means do a web survey next to put those %s in place once you know the statements to put the %s with.

Market Research: Quantitative or Qualitative?

by

Scott Anthony recently advised, “In Market Research, Use Numbers with Caution.” He added, “Companies too frequently default to quantitative research because they think there is safety in numbers. It’s a lot easier to justify a strategy by saying, ‘The data suggests’ than by saying, ‘My intuition suggests.’ But sometimes numbers provide false confidence and obscure real opportunity.”

Anthony’s point of view is quite different than the point of view presented by Robb Mandelbaum, who recently said in Inc., “Given limited resources … it generally makes sense to go quantitative.”

Both articles are excellent  – and both present very different points of view.

So how do you choose between quantitative (surveys) and qualitative (focus groups, in-depth interviews) when performing market research projects with technology participants?

Sometimes, quantitative (surveys) research is better

1. When you need data to support a claim with investors, press, or internal stakeholders.  Dimensional Research has done a number of Web surveys that our customers have used for PR purposes, including these recent ones on anti-virus and desktop power management.

2. For trending purposes, quantitative studies are also the best.  Dimensional Research has a number of clients that follow the “Would you recommend this company to a friend?”  question promoted by HBR, and watch the responses to that trend over time.  It’s a great way to track trends in customer feedback.

3. Of course, if time is the greatest consideration and you need some kind of quick feedback – Web surveys have a big advantage.  The are FAST.

Sometimes,  qualitative (focus groups/in-depth interviews) research is better

Numbers can be deceiving and there is no better way to find that out than to talk live to people who give you numbers.  I was recently doing a competitive study, speaking to end users of a client’s competitor’s product.  My client, as is natural, was most interested in the negative feedback about the product.

So to introduce the topic of what was good and bad in the product, we started by asking the customers to rate the product being discussed on a scale of 1-5.  And of course, asked the important market research follow up question, “why?”

I was consistently surprised by users who had raved about the product, saying it was perfect and there was nothing they would change about it given the chance – and they’d assign a 3 out of 5.  And other users who basically spent the whole conversation whining and complaining about the product would give it a 5!

Numeric scales can be misleading and in these cases, qualitative studies such as focus groups or interviews are better.  I would always recommend qualitative studies when you are looking for thoughtful answers including:

–          Messaging validation for products that are new to the market

–          Market validation

–          Understanding objections and barriers

–          Product feedback for enterprise products (web surveys do a better job with consumer products that are simpler to understand)

Qualitative vs. quantitative is a good conversation to have with your market research provider – although of course you should understand their expertise. If you only have a hammer every problem is a nail, so expect a quantitative-focused research house to tell you to survey thousands of prospects, and someone who only does focus groups to emphasize qualitative approaches.

A good research firm will know if they don’t have a fit with your needs and will point you in a better direction.  For example, Dimensional Research does not do conjoint analysis, but we have a great partner that we can refer you to if that’s what you need.