Dimensional Research

Technology Market Research

In-depth Interviews or Focus Groups: A Reprise

by

Ever since I published In-Depth Interviews vs. Focus Groups, almost two years ago, it has been our most visited blog post. Not only is it the most visited post of all time, it has also been the most visited article on this blog every month since it was published.

It seems to be pretty obvious that people are confused about the difference between the two most basic methodologies for conducting qualitative market research. As researchers, we continuously execute both types of research approaches and instinctively understand which will be best to meet the goals of a  particular project. But we need to be very clear with clients who are not research savvy.

I frequently find myself having these types of conversations with clients. I often get requests asking for a quote for a certain number of focus groups, but when we review the goals of the project, it may turn out there is an opportunity for much better results given the same budget if we were to do interviews instead – especially with competitive research or message validation.

I’ve learned a couple of things:

1. Don’t be afraid to suggest an alternative to a client if it’s right for the project.

Your job is to be an expert and provide the right research solution, not just deliver against the tactics of a client’s request.

2. Be clear and use simple terms when explaining research methodologies.

It’s easy to lapse into “researcher lingo,” but clients will resist alternates if they don’t understand them.

3. Use your best researcher listening skills to figure out why the client is asking for a specific approach.

Clients frequently don’t know alternatives exist. They may have observed a focus group once and saw good results and are asking for that because they don’t realize there are other approaches. It’s also typical that there is an unstated goal to expose a particularly difficult stakeholder to direct market feedback, in which case a focus group may be the right choice, even though other indicators would suggest in-depth interviews.

Controlling the Impact of Observation

by

This tweet made me smile:

It was very funny, and anyone who has sat on the other side of a customer service call probably wishes for more “big bosses” to observe. But as a researcher I also found this to be a classic example of how you have to be very careful, when conducting market research, that you don’t influence behaviors just by being there. Clearly this boss still doesn’t really know what his staff is doing.

Of course researchers have all sorts of tricks for managing this dynamic, depending on the scenario. In my experience, the most important thing in preventing the impact of observation is to be aware that it happens. Simply asking, “What exists in this research environment that may influence behavior?” goes a long way.

I don’t know the background of @BAbackpacker’s situation, but if the “big boss” really wanted to understand the customer service dynamic, he could have recorded a wide range of phone conversations without letting the staffer know which specific ones would be reviewed or listened in, using a technology that was less obvious. Of course he could also hire a professional researcher who would be able to analyze the situation and create a more natural environment.

Web Surveys: How Many Questions Can I Ask?

by

There is no “right” number of questions to ask in a Web survey, but here are a few guiding principles.

You can include a lot of easy questions

It takes very little thought to answer questions about your gender, age, and country of residence.  It may take a second more to consider whether you are willing to share your total household income.  And you know really fast if you have bought a particular product or have called their tech support in the past week.

Just keep the options simple. For example, listing every possible country in the world in a drop-down list is complicated so consider asking about regions instead if the project goals allow.  It also helps to put these kind of easy questions on a single page so there isn’t time spent waiting for pages to load.

Use rating and ranking questions with caution

Markettools did a great webinar a while back.  (I can’t find the link unfortunately, if someone recognizes this study please send to me and I’ll add the link).  They analyzed the behavior of thousands of survey takers to see where drop-off happened.  The glaringly obvious finding was that rating and ranking questions  (those huge matrices with lots and lots of boxes to check) make survey takers go away.

Obviously there are times when research goals demand a rating/ranking approach, but do your best to limit your use of those types of questions when another approach will work. A checkbox series may be simpler to complete even though it is actually more questions.  Minimizing the number of options in a rating/ranking question also helps.

Consider the motivation of your audience

Customers who are giving feedback on a product they use on a regular basis will have a lot of tolerance for a large number of questions and in fact may welcome the opportunity to answer many questions that will simplify their use of your product in the long term.

We’ve done very successful customer surveys that offered no reward and took 20-30 minutes to complete.  The audience was vested in the topic and was engaged.  On the other hand, we often do Web surveys to gauge attitudes of IT professionals. In this case we offer a copy of the final report. We keep these surveys to less than 20 questions with at most one rating/ranking question and one open-ended question.  It takes about 3-5 minutes for a typical IT professional to do one of those kinds of surveys, which seems to match their motivation level to get the report.

The Metric that Matters: Time to complete

If your metric for building surveys is number of questions, you can easily end up with a complicated survey that will give terrible results because it’s confusing for participants and impossible to analyze.  Instead track the time to complete the survey.

Get somebody to take the survey who has not been involved in the development but knows the topic and see how long it takes them.  You might be surprised to learn that your “3 minute survey” actually takes 15 minutes. Much better to find out before you field rather than later, when nobody completes the survey, or you get comments about survey length instead of value-added feedback.

Customers Don’t Talk Like Marketers Do

by

One of the important jobs of researchers is to “translate” between the lingo of marketing people and the way that their customers talk.  It’s an obvious thing when you say it, but a good researcher makes sure they are constantly doing a sanity check and purging “lingo” from their conversations.

I was reminded again of this basic concept on a recent project.  We were hired by a client’s Web team to learn about what type of marketing assets were most compelling for a particular kind of audience. It was a great research project – a clearly defined problem, with a clearly defined target audience.

Customers don’t think like marketers. When an IT professional goes out to get information, they don’t really think too hard if it’s a Web page, a PDF,  a video, or a even a call from their vendor’s account manager. They think about getting an answer to their problem. IT professionals don’t think of “consuming assets”, many of them don’t even know what a marketing asset is.  They just think they “get information from their vendor.”

By asking questions that used their own words to talk about real experiences (for example, “The last time you purchased this type of product, did you visit the vendor’s Web site?”) or showing them examples of specific Web sites and seeing what they’d be most likely to do, we talked to them in their own words and got great, usable answers.

The final step was to translate that back into the client’s marketing speak, because of course they needed to know if they should develop a video or a white paper or an interactive assessment or….

Do You NEED More Data?

by

I am a huge fan of Patty Azzarello. Her blog on business leadership is consistently full of the kind of advice that makes you change the way you work (or wish you’d had someone to tell you before you learned the hard and painful way.)

She recently posted a great article, titled “Do We Need More Data?” It’s a great question and one that market researchers are often scared to ask, because if the answer is “no”, you lose business.

She makes a very important observation: There is time for market analysis and study, and there are times when either you know the answer, or there is no more useful data to be had.

Patty is not bashing market research here, the way some bloggers often do. She merely points out that it’s worthwhile checking in to make sure that’s what your project needs before you move forward.

She also suggests a great question that I ask all my clients: “What additional data is available that’s going to help me?” Going into a research project, you should have a good idea of how you will be able to act on what you learn. If your research shows that the market has a strong need for something that is outside of the core competency of your business, you may have done something interesting, but you haven’t added value to your business.

As always, it’s about goals. And when knowing specific information about the market is going to drive better business decisions, then you do need more data, and it’s time for a market research project.

Market Research: Segmentation Matters

by

As a mother of two girls, I absolutely loved this clever post by Julie Kurd:  Fame or shame…It’s all in the industry (or the household) and her example of how the same mom that is “fabulous” to a pre-schooler is “frustrating” to a teenager.

Aside from being funny and brilliant, this post also drives home an extremely important point about market research segmentation – especially with technology purchasers.  Too often we think of segmentation as being about role, geography, or industry only.  In technology, I would argue that maturity is an even more important segment to consider.

Those same behaviors that Early Adopters love about you – your deep technical descriptions of everything, your willingness to change course on product development quickly, your frequent code releases – will drive most of your early majority and definitely all the late majority CRAZY!!!  That same behavior that you’ve been getting positive customer feedback on for years is now generating negative feedback.  Go figure?

So, the question is, can you recruit Early Adopters? It’s definitely tricky because of course people don’t identify themselves that way.  For some areas it is a straightforward question – for example, if someone hasn’t done any virtualization yet in 2010, they’re definitely late majority or laggard.  If they’ve already implemented a private cloud in production – clear early adopter or possibly even innovator.

But in most cases, it’s not that clear.  An experienced moderator/interviewer can do some important segmentation in the analysis and credit different behaviors and feedback to maturity levels, helping you find and recruit early adopters.

Do-It-Yourself Research

by

As a technology-centric market research firm, our competitors aren’t usually another market research firm – very few have the experience with corporate IT that we do.  Our #1 competitor is “We know our customers, we don’t need a research firm.”

We’ll be the first to say that a company does not need to hire an external agency to do market research. (I’ll pause for a moment so all the professional market researchers can catch their breath at the blasphemy.)

Back when I had my career on the “other side of the glass,” I made it a point to talk to my customers regularly.  I’m a big fan of the Crossing the Chasm model of having the product marketing manager run the beta.  I always found that while my R&D counterpart did his magic to get a new software product to actually work, I could chat up the user and find out a lot about their pain, and why they were bothering to spend the time to deal with a product that was so buggy it needed a PhD to install it.

There are many benefits to “internal research.” It doesn’t have to be formal.  Go to a trade show and give your elevator pitch to 100 people in one day and see how many get it – you will learn a LOT from the questions they ask in response to the pitch.  Invite 5 users to a 1-hour phone call and ask them a few questions and really listen to what they have to say to each other.  Take a customer out for dinner and hear what they have to say outside of the confines of the office.  You will learn an incredible amount of important information from any of these activities.  And in fact, because of your expertise, you may get even more from these exchanges than an external researcher would.

The trick is to figure out when an external perspective is going to add additional benefit.  There are three areas where external resources can add real value:

1. Just Listening

I was presenting a customer report on product ROI and the client – a really talented product marketing manager – asked, “how do you get them to tell you that stuff?”  The problem with being an internal stakeholder is that you are expected to have the answers.  Which means that any customer interaction will be an exchange of them giving you information and you dealing with their concerns.  As researchers, we can gather input, but since we don’t really know the answers to the questions, we don’t have to defend or position anything.  We have the luxury of just listening to the feedback and drilling in to really understand it, without needing to come up with the answer on the spot like an internal stakeholder is expected to.

2. Neutrality

As researchers, we have no vested interest in the outcome of a research.  We have never fought for a particular point of view.  Good researchers never develop preconceived ideas before the research is executed.  External research (conducted well of course) is absolutely the most neutral form of feedback you can get.

3. Finding Non-Customers

Researchers have access to all kinds of people that you may not – people who have never heard of your company, competitor’s customers, prospects in a new vertical industry.  If it can be profiled and there are enough people in the world that match that profile, external researchers can find them and get them to talk to us when you can’t yourself. (Actually sometimes it’s even easier for us to find customers, depending on how bad the client’s customer database is – an ongoing problem with enterprise technology companies in spite of the improvements in CRM in the last several years.)

Communicating Research Findings: Three Personas to Watch For

by

The real result of any research project is not the final report – it’s the growth that happens as a result of business owners acting on the findings.  Too often, great research ends up shelved because there is no action – or even outright disbelief.  As researchers, it’s our job to present our findings so that they influence future decision making.  Of course, working with your stakeholders to understand any preconceived ideas heading into the research will help to frame the discussion, but you’ll also want to understand the personas that will be consuming and acting on the research.

We have found that there are three types of research consumers that need special attention. Each should be communicated with differently. Knowing which one you have in the room will help considerably with presenting your findings in a way that would lead into action rather than disbelief and resistance.

1. The Impatient Executive

This is the guy with very little time and strongly preconceived ideas.  This individual is very smart – but be careful, they are VERY confident in their view of the world.  Communicating with this person is all about the executive summary.  Make sure you get to your 3-4 top takeaways within the first 5 minutes of the meeting.  Then watch carefully for any reaction.  If your research matches their beliefs, no problem.  Present the Executive Summary, they’ll agree, and will very likely leave the room early, happy that the rest of their team is getting a good validation of what is going on.

But keep an eye out for the place in the executive summary where the eyebrows come together, the blackberry is put down, or the ever-so-slight frown appears, then make sure you spend a little extra time on that area. Put a special emphasis on the anecdotes picked up in the research for that topic.  Share the one opinion that was the outlier and then frame that as the clear exception.

2. The Data Lover

This persona is the researcher’s best friend.  They will love you and your findings, and will spend hours drilling down into the subtle details of the findings.  The trick with this persona is to make sure they don’t derail the meeting with minutia that will bore the other attendees, or even worse, prevent you from getting to the real meat of the study.  If you know you have one of these people, a pre-brief with just them is a good idea.  Offering to “Take it offline to do the deep dive in this subject” is also an option – one that will be deeply appreciated by the Impatient Executive.

3. The Skeptic

This persona simply doesn’t believe in research – any research.  Their belief in the way the world works is based on years of experience, and a fact or two isn’t going to override that. The trick to communicating with this persona is to start with the parts of the project that do match their world view.  If you can lead off with one or two things that the skeptic agrees with, they’ll have more faith in the research and be more open to hearing the next point that is out of line with their beliefs.

Of course, that still leaves you with the most challenging scenario – the meeting with a big group of people including all of these personas, plus a few more.  Our best advice – do great research so you are confident in the meeting; get that executive summary in early; ask for questions to be held until the end if you’re getting off schedule;  and then do your best to be dynamic and engaging.

Yet Another Useless Customer Survey

by

I’m on the phone right now with my bank, doing a “post transaction” survey.  I always take these – what researcher doesn’t? And once again I’m annoyed.  What has my bank learned from doing this survey?  As far as I can tell, ABSOLUTELY NOTHING.

The purpose of my call to the bank was to get that number you need so someone can wire you money.  Let’s see how the “post transaction” survey went:

  • A question about the banker I talked to.  He was just lovely.  I asked my question badly, because I forgot the name for that number (routing? IBAN? something like that), but got an answer right away.  Gave him a high number.  Good so far.
  • Wait, aren’t you going to ask me about the experience of getting to my lovely banker?  Do you care that it took three calls and five minutes to get to the 30 second conversation?  Apparently not.  No questions about that.
  • Now a question about the range of services offered by my banker.  No idea.  I only asked him one question.  He answered it well.  I can’t answer this question, but “don’t know”  is not an option on this survey. I can only choose “1=unacceptable through 5=excellent.”  Where’s the N/A?  Do I say “excellent” since I liked him and don’t want to get him into trouble?  What if his bonus is based on this survey?  But that will be misleading in the analysis.  I could give a 3 to at least not skew the averages much, or just hang up.  Or do I give him a 5 because I liked that he didn’t try to sell me anything else after he answered my question?   I’ll just give a 4 to split the difference.
  • What?  We’re done? Where is my “If you’d like to leave any other comments please visit www.bank.com/survey” so I can tell them about the last question.  How difficult is it to have a form for follow up in this day and age?

I will stop now before I begin ranting, but once again I feel like today’s ease of implementing surveys on web sites and telephone is causing some of the worst market research that has ever happened, and in the process annoying people who might stop doing surveys that are actually well designed and will drive better business practices.

A Researcher’s Confession

by

I admit it.  Although web surveys are one of our most popular research offerings, I strongly prefer qualitative research (focus groups, in-depth interviews).

I can’t help looking at the findings in web surveys and feel like I want to dig in and ask more questions.  The participants say things that don’t make sense to me and I want to know why.  Or they make short comments responding to open-ended questions that leave me with a dozen follow-on questions I don’t have the opportunity to ask.

When I present qualitative findings, I know I can answer any question that comes up with complete confidence.  With quantitative findings, I always know that there will be questions where the answer is, “We can’t draw any conclusions based on this survey.”

This came up again last earlier this month:  We just did a series of in-depth interviews where we wanted to understand perceptions about the cost of various alternatives.  All the participants in this study had identified themselves as product decision makers who had full visibility into costs – a requirement for the study.   If you had just looked at the first level of answers people gave, you would have thought that one of the tools we were looking at was very inexpensive compared to other options.

But because this was an in-depth interview, I got to ask that all-important “why” question. I quickly realized that while all the participants had been educated on the “line 3” costs that were billed directly to their organization, not everyone was aware of the additional “line 10” costs that had to be added to support this different approach.  When you added both of those up, the tool that originally appeared less expensive turned out to have a similar TCO to other options.

Now, it’s true that we could have found this out by writing a good web survey, but one of the secrets to writing great web surveys is to know the answers to all the questions first.  We continue to recommend web surveys as good vehicles for quantifying concepts that you know well, but want to put an accurate % by each of the options.  This is a valuable thing to do, especially for market sizing, external marketing and PR purpose.

But for finding out the answers that you don’t really know, start with qualitative research – and by all means do a web survey next to put those %s in place once you know the statements to put the %s with.