Dimensional Research

Technology Market Research

Always Ask the Question

by

Methodology matters. Last night I had one of those moments that reminded me again that a structured research interview is not just a conversation.

I was on the phone with an IT manager talking about a product he had used for about a year. His experiences were bad. He had harsh things to say about the product, the implementation, and the lack of business results. He had been  articulate and forceful about his frustrations, so when I got to the “would you ever recommend or purchase this product again” question I was confident the answer would be a resounding “NO! NEVER! NOT EVER!”.

I was wrong. In fact, my participant had changed jobs in the past two weeks and he had already spoken to his new management about purchasing the product – the exact same terrible one we’d been discussing. I was really surprised and asked again to make sure I’d understood. It turned out that while he didn’t like the product and thought it was terrible, it was still much better than nothing. But I needed to ask the question to figure that out.

In a social conversation, it’s ok to project and draw conclusions. You end up with a more interesting exchange. But in a research interview, you can’t. It’s too important to understand all angles.  So always ask the question, even if you are confident you know the answer. 

Are Your Research Participants Smart?

by

When doing research, understanding if participants are really experts in their field is important. You have to know if their feedback is coming from a place of knowledge, one of assumption, or perhaps even ignorance.

This can be tricky. There is a type of person who is incredibly good at sounding knowledgeable. They can drop clever sound bites or smiling knowingly to give the impression of deep knowledge, even if they don’t have any.

This is particularly important in our field of IT and business technology market research where many of the concepts are very complex. Even qualified participants who have achieved amazing career success may not know what they’re talking about once you get into lower-level details. This is especially difficult with senior participants who are removed from the actual day-to-day activities, but are hesitant to use those important words “I don’t know”.

Research projects must include perspectives from all kinds of buyers who influence purchasing decisions – which usually includes both “smart” and “not smart” participants. The challenge is two-fold:

  1. Getting a mix of participants that represent the knowledge in your market
  2. Knowing which is which when you present your analysis

Because, here’s the thing. Taking “sounding smart” input and presenting it as “smart” input will not drive business results.

Here’s a few tips for managing this dynamic in research projects:

  • Be “smart” yourself.  You have to have some depth of experience with your topic to know when you’re talking to someone who isn’t an expert. Do your homework prior to any research conversation.
  • Give participants permission to say they don’t know.  Do this directly. Say something like “This is market research – you are not being tested.  It’s us who are being tested. If people like you aren’t aware of a concept that is actually the most important thing for us to learn from this project. Feel free to say you don’t know. That’s a great answer.”  Then praise them when they say  they don’t know.
  • Add a “testing” question.  It may not be the goal of the project, but adding at least one question that’s a bit “in the weeds” and seeing how that is answered will allow you to rate the participant. You should frame it as that though “not to get into the weeds since this isn’t the goal of this project, but I am curious what you think about…”

Market Research and “Ethical Treatment of Human Subjects”

by

I don’t know the AAPRO well enough to know if they were trying to be funny, but I laughed at the language they used to write this survey particular tip:  Maximize cooperation or response rates within the limits of ethical treatment of human subjects.

(Thanks to Cathy Harrison and  Bernie Malinoff for Tweeting this article.)

Medical and psychological researchers have strict ethical guidelines in place dealing with people in their research, maybe people who run Web surveys need some also.  Here is our input into guidelines that should be included in a manifesto for being kind to human participants in market research studies.

  1. Never force someone to give you a wrong answer – When Steve Carrell gave Conan O’Brien his exit interview on the Tonight Show and asked if working at NBC was “great”, “really great” or “fantastic”, I had a flashback to real-life employee survey I had to take years ago.  Web surveys should always cover every possible option someone could give or offer an “other” option.  A good survey will have a comments area at the end to allow users to clarify anything they felt didn’t represent their input.
  2. Use complex ranking or rating matrices with caution – It is just plain painful to answer long rating or ranking questions even when you deeply care about the subject. You always end up missing at least one line and get that oh-so-not-helpful “please answer number 14”.  Be kind and use matrices sparingly and only when there are no other approaches to get the answer you’re looking for.
  3. Don’t ask questions that simply don’t matter – Seriously, does my ISP need to know if I’m a man or a woman?  Is that going to change the way that they deliver service to me?  If it does, then by all means ask, but I frequently feel like somebody took a market research 101 class back in their first year of college and ever since have opened with demographic questions on age, gender, and ethnicity independent of if it matters to the survey goals.  There is something to be said for asking “warm-up” questions that are easy to answer, but surely there is something more interesting then gender to ask.  [UPDATE:  Of course there is always the “Elinor Exception” to this rule as rightly pointed out in the comments.  If you need to add a sanity check on the quality of the sample, then go ahead and ask the gender question.]
  4. Don’t expect participants to know all your lingo – Seriously.  Speak plain English  (or French or Spanish or Chinese or Japanese or whatever language your participants do).  Don’t throw in tons of jargon without defining it first.
  5. Be realistic about how long your research takes – It seems as if every survey in the world is “10 minutes” long.  Surely you can have at least one person who didn’t write the survey questions try it out and tell you how long it realistically takes an average person.

I’m sure at Dimensional Research we have on occasion broken one of these rules, but we try our hardest to ensure our participants don’t suffer in the process of giving their feedback.  We’re always happiest if they actually enjoy it.

What would you add to a “Manifesto on Ethical Treatment of Survey Takers”?

Market Research: Guiding Responses

by

In this blog, we’ve talked a lot about market research question design, and for a good reason. When preparing a market research project, it’s important to understand how you’re leading your audience. Of course you need to guide discussions and probe for specific responses, but you need to be very aware of what you’re doing and how the way you ask influences the responses.

A very simple example of a subtle guidance: When taking personality tests, if you say “take your time” you get different responses than if you say “do your best”. “Take your time” implies that you want very thoughtful answers, and people respond accordingly. “Do your best” is neutral and doesn’t have  implicit guidance built in.

Think about your research goals, and set up your questions appropriately.

If you’re doing a message test where in real life people don’t take a lot of time to think through underlying ideas, create a similar environment with guidance like “Please review and give me your first gut reaction” or “Don’t over-think this, I’m looking for your initial response.”

If you’re looking to understand pain, it may take some digging to get to the source because your participant may not actually be aware of what’s going on.  This is particularly true with technology professionals that have lived with the pain for a while and have a workaround in place. They may not notice how much time they spend doing something that is not value added because it’s just the way it works.  In that case, ask questions that encourage  thoughtfulness, such as “Why is that?” or “Really, tell me more about that” or “Could a different approach be more valuable?“

Market Research: Can you trust it?

by

Hal Varian, chief economist at Google recently said that the “the sexy job in the next 10 years will be statisticians”.

I think he’s right.  Every participant in a first-world economy should have a solid understanding of stats.  Not that I’m recommending that everyone become a statistician, but with the volume of information out there, it’s important to understand how data can be used to sway you.

Here are two “statistics” about my own life:

i) I was the only female graduate in Pure Mathematics at my university in my year  (true).

ii) 50% of my graduating class who majored in Pure Mathematics were women (also true).

One of those statements paints a picture of a mathematics education that is oppressive to women, somehow subtly driving females away.  The other paints a picture of a very progressive math department that gave women the same opportunities as men.

As I’m sure you’ve figured out, there were only two people in my graduating class that majored in Pure Mathematics.  But without that information, I could have easily mislead you. Clearly the statistical significance is not there.

When someone presents information, whether a market researcher or anyone else, always think about the reliability. What’s the methodology?  How big is the data set? Is the audience who completed the research the right audience to comment on the topic?  Is it representative?

Research bias is very real. You need to be aware of this and always ask the right questions in order to determine whether information presented to you is valid – whether selling enterprise software or listening to the media.

Dealing with Negative Feedback

by

Negative feedback is good for you. 

Seriously.

While it’s very human to only want to hear how wonderful your solution is, you need to learn about any objections to your product NOW. Not later, when you have to deal with an entire slipped pipeline, because you didn’t realize there was a deal-breaker.

You need honest feedback, and you need it now.

But how do you ensure feedback that is real? Very few people, no matter how stupid they think your idea is, will actually tell you that. You have to create an environment that allows people to tell you about the downside. 

One of the more painful exercises you can go through is to put a group of people together in a focus group and ask “what DON’T you like about this solution?” Once you ask them this question, and give them permission to be brutally honest, they WILL tell you about everything they don’t like. And the list of things they don’t like will be very long. And their list will inspire the other participants to think about even more things that they don’t like.

But as I said, this is GOOD FOR YOU!  If you know about the objections, you can prepare for them in advance. In the long run, this will save you significant amounts of time and money.

Market Research: Getting to “No”

by

One of the most important skills in moderating any market research discussion is to make sure you’re not “guiding the witness” to tell you something that is not accurate.

I’ve discussed here the importance of asking market research participants open-ended questions such as “can you tell me more about that?” and “is there anything else?”  I also written about the importance of getting honest feedback, and suggested ways to make sure you do.

Today I’d like to emphasize the importance of giving the participants permission to NOT know.

Market research participants are smart, competent people who know what is going on in their environments. It’s natural that they’re only human and don’t want to come across as unknowledgeable. Nobody wants to admit, “I don’t know.” 

However, YOU need to know the truth. If they don’t know about a product, you need to know about that and adjust your marketing efforts accordingly. It’s just as important for you to uncover that participants are unfamiliar with a product or an idea as it is to know their opinion about it.

The key is to ask questions that allow a participant to tell you the truth without feeling like an idiot.

A question that is in many research guides is an awareness question. “Are you familiar with product X?  What do you know about it?”  I don’t like asking the question this way. It almost guarantees that some selection of the participants will answer, “Yes, I’ve heard of it. I read about it in a magazine” just so they don’t have to admit they have never heard about the product.

Instead, ask “Are you familiar with product X – and please feel free to say no, that’s just as important for me to know.” This will give the participant permission to not know without feeling stupid. 

Another option is to ask, “What do you know about the following product – if anything?” that also leaves the door open for participants to admit they don’t know without worrying about disappointing you.

Market Research: It’s All About Listening

by

When you don’t listen to your customers or prospects, you risk losing them.

According to a recent report, “Did not listen to me” is the most widely experienced problem faced by 38% of services buyers. Additionally, 55% of buyers said they would be “much more likely” to consider hiring the provider “if they listened better.”  As Drew McLellan points out, if you add up the “didn’t listen” and the “talked too much,” that’s 63%.  Two thirds of your sales calls are being wasted simply because you talk too much and don’t listen enough! 

Market research is all about listening. It is also about asking the right kind of questions so that you’ll get the right kind of feedback. 

Here are three tried and true tips that I always use in my market research to make sure I actually listen to participants. These tips are applicable to every area of customer and prospect interaction.

  1. Shoot for the 80/20 rule.  Always ask yourself, what percentage of the time are you talking, and what percentage of the time are the participants talking?  I shoot for the 80/20 rule.  If I’m doing more than 20% of the talking, then I’m probably not really listening.
  2. Ask questions that encourage talking – not just “yes/no”.  Are you telling the person what their pain is, or are you letting them tell you about their pain?  A good researcher doesn’t say, “here is your pain – isn’t that right?”  They ask about pain to see if the pain they believe the prospect has actually comes up as a priority.  Of course you can always guide a discussion to keep it relevant, but you need to ask open questions in order to get real feedback – and give customers and prospects a real chance to say “no, I don’t care about that”.
  3. Look for what you don’t know.  Never ask a participant “Do you agree?” 90% of the time they’ll say yes, because that’s the easy and polite thing to do. Your goal is to create an opportunity to hear something you DON’T know. 

To quote the Greek philosopher Epictetus, “we were born with two ears and one mouth, so we can listen twice as much as we speak.”  Thanks for the inspiration Drew.

Market Research: Ensuring Honest Feedback

by

I’ve sat through many advisory board and sales meetings with customers who have just been wined and dined. Then, during the formal part of the meeting, a product manager presents a new roadmap presentation and wraps up by asking, “what do you think?” with a face that clearly shows how proud they are of their work. 

You are very, very fortunate if you have customers that will actually give real, honest feedback in a scenario like that. They want to keep things positive and upbeat – they just saw pictures of that product manager’s kids or heard about their latest adventure trip. They don’t want to get all negative, even if that is what they really think.

One of the things that market research does really well is to intentionally CREATE an environment for honest feedback and for uncovering information you haven’t anticipated.  To create such an environment, you need to:

            1) Make sure the conversation is not about the personal relationship. Of course you want the relationship, and you want to leverage that. But the feedback can’t be influenced by whether someone likes you or not.  Of course they like you – you’re a tremendously likable person!  But the feedback must be about the topic, the market, and the pain points, not about what they think about you. 

You need to use verbal and non-verbal cues to set up the discussion so that it is open, non-judgmental, and the client is never aware that what they’re saying is bad news for you. You need to get the bad news now – not after you’ve been through a long, involved sales cycle that ends because there is “no budget,” which really means “not enough pain.”

            2) Dig for negative feedback.  Don’t just ask, “what do you think?”  Encourage people to talk about what they DON’T like.  You can’t make it better if you don’t know what’s wrong.  Ask, “What do you LIKE about what I just showed you?” then ask “What DON’T you like about what I showed you?” 

            3) Put feedback in context.  It’s not enough for a user to connect with the pain you solve – it has to be a priority!  I recently was in a focus group where a participant described a situation troubleshooting a particularly elusive problem in a complex application. It was a dramatic story with dozens of IT staff from different technology groups getting on bridge calls, pointing fingers, escalating issues, and finally getting the CIO involved in a 48-hour saga that ended up to be a rather minor configuration issue. I could feel the client behind the glass get excited because this was exactly the type of problem they envisioned their solution solving.

But then we asked how important it was to solve this kind of issue so it didn’t happen again. The participant explained that it wasn’t important at all.  It was the first time in 15 years it had happened and they had solved it.  They had moved on.  The pain wasn’t a priority.  Fortunately that group did express other pains that the client could solve, and they learned important lessons on ways of positioning their product that matched the most important pains.

Employing these techniques a great way to encourage your customers to give you real, honest feedback and to help you uncover pains you were unaware of.

Market Research: Listening for the Unexpected

by

Most businesses have an existing product or market that they are targeting, and their goal is to learn how to serve that marketplace more effectively. Any reasonable researcher should know that if you have a virtualization management solution, it isn’t that useful to talk about HR problems.  But while you need to stay on topic, you also don’t want to miss the chance to uncover information you are unaware of and may be a huge opportunity or a weakness you must address. This is one of the real advantages of qualitative market research where you can spend time deeply understanding the nuances of customer input.

Treat “unprompted” feedback with special importance.

Sometimes the information you don’t know that you need to ask about, is the most important information to hear.  Of course research must have a goal – but work with your researcher so they know your business and can use their judgment to understand what threads are worth digging into and what will take them into a rat hole that does not forward your business. 

The most important tool for unprompted feedback is this question:  “Is there anything else?”  This is the final question Dimensional Research uses for any market research interaction. After you’ve asked all your carefully crafted and specific questions, closing with an open-ended question can yield incredibly useful information.

Of course, this is a very important balancing act and research projects need to focus on their goals – not interesting side topics.  The core questions must be asked. But Dimensional Research makes a point of going through all input from every project with a special eye for the unprompted information.  Most of it is one-offs, with each participant having a special thing that only they are interested in.  But often you can spot trends in unprompted information that are really important. 

We’ve done projects where it turns out there was a critical development tool that needed to be integrated with the client’s product – not something they had realized was vital until the majority of participants talked about it, unprompted, in a series of customer and prospect focus groups.

In another project we didn’t realize that customer loyalty was almost non-existent, until we went through the unprompted feedback from interviews with the highly satisfied, happiest customers, and discovered they regularly evaluated the competition – not a question that we asked about specifically.

So do stay on topic.  But keep your ears open for the “+1” opportunities or customer issues you hadn’t recognized – especially if you hear them from multiple sources.  Possibly minor product changes or messaging tweaks can help you address a different pain, target a new and lucrative market, or address important weaknesses.