Dimensional Research

Technology Market Research

Bad In-depth Interview

by

It’s very common to be invited to participate in a bad web survey or a bad phone survey. But this week I had the dubious honor of being a participant in a truly bad in-depth interview. I had been invited to give my perspective on the use of research during an enterprise sales cycle.

Usually researchers are automatically excluded from these kinds of research projects. However, Dimensional Research develops research reports including customer-based ROI analysis and web surveys, so I was asked to participate and couldn’t give up the opportunity.

Here’s just a few of the things that went badly during the 45-minute telephone conversation:

1) The call opened with the interviewer telling me that he “was convinced there was a huge gap” and he was doing the research to prove it. Yes – he started out by stating his conclusion and his bias!

2) After a 10 minute preamble, during which time I was easily distracted by a client skype-ing me, he finally asked me the first question. His opener, which sets the tone for the entire conversation was this gem: “Where are you located?” They already had taken my contact information previously, so it surprised me that they opened with this.

Let me clarify – I completely understand a need to confirm information gathered previously, but this seemed like a weak opener. Personally, I prefer starting an in-depth interview asking the person to tell me about whatever their involvement is in the topic of the study. That usually covers a lot of the basics (job title, location, etc.) without asking the boring laundry list of questions. Of course you can always ask those questions later if they’re not covered in a more engaging fashion.

3) The interviewer had sent a supporting document – probably about as many words in the document as is in this post, in about the same font size – and then proceeded to read the whole thing out loud to me. He didn’t seem to get that I CAN READ and was done with the document way before he finished it and was being distracted by a text message. Again, certainly there are times when it is appropriate to review materials with the individual, but one of the benefits of an in-depth interview is you can get a real feel for the participant and understand when you can move more quickly or need to progress more slowly.

4) The interviewer ARGUED WITH MY ANSWERS!!! Now I know that all researchers need to dig deeper and sometimes playing devil’s advocate is a good tool. We’ll often use questions like “I understand that you prefer this, but can you explain why that isn’t a preference?” But this guy actually said “Really? That’s odd. In my experience it never happens that way.” Talk about not feeling valued!

5) The interviewer did finally let me get a word in edgewise, and I was half-way through an idea when I got the “Thank you Diane, this was great” and we were done.

I will confess I was a bad research participant and didn’t turn off my instant messaging, email, and other distractions. But I was taking time out of my schedule to share some expertise that I had and the researcher didn’t, and there was no attempt made to make me feel heard or make the experience a positive one for me.

Such a waste of 45 minutes – for both of us.

In-depth Interviews or Focus Groups: A Reprise

by

Ever since I published In-Depth Interviews vs. Focus Groups, almost two years ago, it has been our most visited blog post. Not only is it the most visited post of all time, it has also been the most visited article on this blog every month since it was published.

It seems to be pretty obvious that people are confused about the difference between the two most basic methodologies for conducting qualitative market research. As researchers, we continuously execute both types of research approaches and instinctively understand which will be best to meet the goals of a  particular project. But we need to be very clear with clients who are not research savvy.

I frequently find myself having these types of conversations with clients. I often get requests asking for a quote for a certain number of focus groups, but when we review the goals of the project, it may turn out there is an opportunity for much better results given the same budget if we were to do interviews instead – especially with competitive research or message validation.

I’ve learned a couple of things:

1. Don’t be afraid to suggest an alternative to a client if it’s right for the project.

Your job is to be an expert and provide the right research solution, not just deliver against the tactics of a client’s request.

2. Be clear and use simple terms when explaining research methodologies.

It’s easy to lapse into “researcher lingo,” but clients will resist alternates if they don’t understand them.

3. Use your best researcher listening skills to figure out why the client is asking for a specific approach.

Clients frequently don’t know alternatives exist. They may have observed a focus group once and saw good results and are asking for that because they don’t realize there are other approaches. It’s also typical that there is an unstated goal to expose a particularly difficult stakeholder to direct market feedback, in which case a focus group may be the right choice, even though other indicators would suggest in-depth interviews.

A Researcher’s Confession

by

I admit it.  Although web surveys are one of our most popular research offerings, I strongly prefer qualitative research (focus groups, in-depth interviews).

I can’t help looking at the findings in web surveys and feel like I want to dig in and ask more questions.  The participants say things that don’t make sense to me and I want to know why.  Or they make short comments responding to open-ended questions that leave me with a dozen follow-on questions I don’t have the opportunity to ask.

When I present qualitative findings, I know I can answer any question that comes up with complete confidence.  With quantitative findings, I always know that there will be questions where the answer is, “We can’t draw any conclusions based on this survey.”

This came up again last earlier this month:  We just did a series of in-depth interviews where we wanted to understand perceptions about the cost of various alternatives.  All the participants in this study had identified themselves as product decision makers who had full visibility into costs – a requirement for the study.   If you had just looked at the first level of answers people gave, you would have thought that one of the tools we were looking at was very inexpensive compared to other options.

But because this was an in-depth interview, I got to ask that all-important “why” question. I quickly realized that while all the participants had been educated on the “line 3” costs that were billed directly to their organization, not everyone was aware of the additional “line 10” costs that had to be added to support this different approach.  When you added both of those up, the tool that originally appeared less expensive turned out to have a similar TCO to other options.

Now, it’s true that we could have found this out by writing a good web survey, but one of the secrets to writing great web surveys is to know the answers to all the questions first.  We continue to recommend web surveys as good vehicles for quantifying concepts that you know well, but want to put an accurate % by each of the options.  This is a valuable thing to do, especially for market sizing, external marketing and PR purpose.

But for finding out the answers that you don’t really know, start with qualitative research – and by all means do a web survey next to put those %s in place once you know the statements to put the %s with.

Phone Surveys or In-Depth Interviews: What’s the Difference?

by

A while back we blogged about phone surveys, and argued that they do not have a place in technology market research. Several comments were made both in the blog comments and to us directly that made us realize that there is confusion about the difference between phone surveys and in-depth interviews (also called IDIs). Here’s how we view this:

Phone surveys and in-depth interviews are similar because:

  • Both are conducted over the phone.

But that’s pretty much the only similarity.

The differences between phone surveys and in-depth interviews:

Phone surveys: A quantitative research method that includes a large numbers of participants.

  • Capture input to a common set of questions with pre-set answer options
  • Administered by a phone survey professional that has a pleasant voice and is trained not to guide the responses of the participants in any way
  • Are usually short: 5-20 minutes as rule of thumb
  • Usually no incentive is given to participants, although there may be a small amount given.

In-Depth-Interviews (IDIs):A qualitative research method that uses a smaller number of highly select participants.

  • Screener is written to allow significant discovery, open-ended questions, and drill-down
  • Administered by a trained moderator who is versed in the client’s business, the goals of the research, the topic of study (in our case technology) as well as techniques for putting the client at ease and getting the kind of feedback desired
  • Usually longer: 30 minutes to 1 hour is common
  • Typically generous incentives are given to participants to compensate them for the time commitment

An easy way to think of the differences are that phone surveys are pretty much like web surveys, but conducted over the phone. Given these definitions, we stick by our earlier recommendation that phone surveys have no place in technology market research. They are significantly more expensive to conduct than web surveys without adding value in a corporate IT study (although agree that there are audiences phone surveys are appropriate for).

That said, there is nothing inherently wrong with phone-based research. In fact, in-depth telephone interviews have become one of our most valuable research methodologies, particularly as clients become more global and travel costs are becoming more of a factor in evaluating research budgets.

In-Depth Interviews, Focus Groups, or Both?

by

When doing qualitative research, we need to decide which is right for the client: in-depth interviews, focus groups, or maybe a combination of both. This depends on the client’s goals.

When are focus groups better?

Dimensional Research recommends focus groups when the client wants to gain multiple perspectives in an interactive group setting.

One of the main benefits of focus groups is that they get the participants brainstorming. When one participant’s comment feeds off of another comment and so on, the group can really dig deep into an issue. When trying to evaluate market acceptance, capture challenges and issues, or understand objections to new technologies or processes, the focus group dynamic is ideal.

Focus groups have another great benefit – the client can sit behind the glass or on a conference call and hear the direct, unfiltered feedback of a large number of participants with no distractions. Focus group sessions are also recorded for further observation.  If your goal is to expose the maximum number of your team to direct input from the market, this is a very efficient way to do it.

When are interviews better?

In-depth, one-on-one interviews with technology professionals can be conducted in person or over the phone. These are appropriate when the client wants to identify detailed perceptions, opinions, beliefs, and attitudes.

In-depth interviews are particularly effective in the following scenarios:

  1. When the client’s goal is to capture feedback on experiences that occur in an isolated way, such as product messaging or product usability testing.
  2. When there is anything sensitive about the feedback that participants may not feel comfortable sharing in front of other people. They may be concerned a competitor is also be attending the focus group. Or in a win/loss analysis, a customer may not feel comfortable sharing details of poor account manager performance if they suspect the rep might be “behind the glass” and could hear him. The customer would feel much more comfortable sharing this information confidentially talking only with the researcher.

Two things that should NOT drive this decision are:

  • Travel – Sometimes the decision is driven by the geographical locations of participants. If you’re doing customer research with customers located in diverse locations or if you want global representation in the groups without the cost of travel, you still have a choice. The one-on-one nature of a phone interviews is an easy option, or for the group dynamic choose online focus groups.
  • Cost – An important consideration, of course, is cost. As a rule of thumb focus groups and interviews cost about the same per participant, so with the exception of travel – not a consideration for phone interviews or online focus groups – cost should not be the driving consideration in choosing the research approach.