Scott Anthony recently advised, “In Market Research, Use Numbers with Caution.” He added, “Companies too frequently default to quantitative research because they think there is safety in numbers. It’s a lot easier to justify a strategy by saying, ‘The data suggests’ than by saying, ‘My intuition suggests.’ But sometimes numbers provide false confidence and obscure real opportunity.”
Anthony’s point of view is quite different than the point of view presented by Robb Mandelbaum, who recently said in Inc., “Given limited resources … it generally makes sense to go quantitative.”
Both articles are excellent – and both present very different points of view.
So how do you choose between quantitative (surveys) and qualitative (focus groups, in-depth interviews) when performing market research projects with technology participants?
Sometimes, quantitative (surveys) research is better
1. When you need data to support a claim with investors, press, or internal stakeholders. Dimensional Research has done a number of Web surveys that our customers have used for PR purposes, including these recent ones on anti-virus and desktop power management.
2. For trending purposes, quantitative studies are also the best. Dimensional Research has a number of clients that follow the “Would you recommend this company to a friend?” question promoted by HBR, and watch the responses to that trend over time. It’s a great way to track trends in customer feedback.
3. Of course, if time is the greatest consideration and you need some kind of quick feedback – Web surveys have a big advantage. The are FAST.
Sometimes, qualitative (focus groups/in-depth interviews) research is better
Numbers can be deceiving and there is no better way to find that out than to talk live to people who give you numbers. I was recently doing a competitive study, speaking to end users of a client’s competitor’s product. My client, as is natural, was most interested in the negative feedback about the product.
So to introduce the topic of what was good and bad in the product, we started by asking the customers to rate the product being discussed on a scale of 1-5. And of course, asked the important market research follow up question, “why?”
I was consistently surprised by users who had raved about the product, saying it was perfect and there was nothing they would change about it given the chance – and they’d assign a 3 out of 5. And other users who basically spent the whole conversation whining and complaining about the product would give it a 5!
Numeric scales can be misleading and in these cases, qualitative studies such as focus groups or interviews are better. I would always recommend qualitative studies when you are looking for thoughtful answers including:
– Messaging validation for products that are new to the market
– Market validation
– Understanding objections and barriers
– Product feedback for enterprise products (web surveys do a better job with consumer products that are simpler to understand)
Qualitative vs. quantitative is a good conversation to have with your market research provider – although of course you should understand their expertise. If you only have a hammer every problem is a nail, so expect a quantitative-focused research house to tell you to survey thousands of prospects, and someone who only does focus groups to emphasize qualitative approaches.
A good research firm will know if they don’t have a fit with your needs and will point you in a better direction. For example, Dimensional Research does not do conjoint analysis, but we have a great partner that we can refer you to if that’s what you need.