IT Participants

Are Your Research Participants Smart?

When doing research, understanding if participants are really experts in their field is important. You have to know if their feedback is coming from a place of knowledge, one of assumption, or perhaps even ignorance.

This can be tricky. There is a type of person who is incredibly good at sounding knowledgeable. They can drop clever sound bites or smiling knowingly to give the impression of deep knowledge, even if they don’t have any.

This is particularly important in our field of IT and business technology market research where many of the concepts are very complex. Even qualified participants who have achieved amazing career success may not know what they’re talking about once you get into lower-level details. This is especially difficult with senior participants who are removed from the actual day-to-day activities, but are hesitant to use those important words “I don’t know”.

Research projects must include perspectives from all kinds of buyers who influence purchasing decisions – which usually includes both “smart” and “not smart” participants. The challenge is two-fold:

Getting a mix of participants that represent the knowledge in your market
Knowing which is which when you present your analysis

Because, here’s the thing. Taking “sounding smart” input and presenting it as “smart” input will not drive business results.

Here’s a few tips for managing this dynamic in research projects:

Be “smart” yourself. You have to have some depth of experience with your topic to know when you’re talking to someone who isn’t an expert. Do your homework prior to any research conversation.
Give participants permission to say they don’t know. Do this directly. Say something like “This is market research – you are not being tested. It’s us who are being tested. If people like you aren’t aware of a concept that is actually the most important thing for us to learn from this project. Feel free to say you don’t know. That’s a great answer.” Then praise them when they say they don’t know.
Add a “testing” question. It may not be the goal of the project, but adding at least one question that’s a bit “in the weeds” and seeing how that is answered will allow you to rate the participant. You should frame it as that though “not to get into the weeds since this isn’t the goal of this project, but I am curious what you think about…”

Three Tips for Effective Win/Loss

by Diane Hagglund

It seems that win/loss analysis is in the air these days. We’ve had a sharp uptick in incoming inquiries for projects analyzing why enterprise software deals are won or lost when there is a competitor in the mix.

The major factor in successful Win/Loss projects is the ability to identify stakeholders who were deeply involved in the process and are willing to talk about their experiences. Here are our top tips to get the right people:

Ensure anonymity – during an enterprise sales process, prospects can develop relationships with their account management team. They may be hesitant to discuss issues that happened during the sales process if they are worried that they’ll be putting someone they know in a bad light.
Expect more effort to secure losses – it’s easier to get people to talk about wins. Once a company has chosen a vendor to have a relationship with, they are more vested in that relationship and willing to put the time into making it work. If a customer has chosen another vendor, they will be less likely to talk to you. Plan for this when building target lists.
Keep it short – Senior IT executives value their time above all else. They are busy. It’s difficult to get an hour of their time, but asking for 15 or 30 minutes feels like much less of a commitment and they will be more likely to participate than if they feel it is a big and onerous conversation. Of course, once they’re on the phone, if they want to keep talking let them! But don’t start with the big ask.

Bad Surveys: Participants Know When Research is Bad!

by Diane Hagglund

A friend of mine who is not involved in the research business sent me this email yesterday.

Hi Diane.

I attended a conference this week and was sent this survey to fill out.

Did you watch the playback of the keynote session?
( )Yes. It was excellent.
( )Yes. It was OK.
( )No, but I plan to.
( )No, and I am not interested.

What I want to know is where is the response “Yes and it sucked!”

Of course this was a bad survey question, which we’ve talked about before (Bad Survey Design; Bad Web Survey Question).

But the broader point isn’t just that the research is bad, but that THE PARTICIPANTS KNOW! They can tell if their option isn’t on the list and they just won’t complete the survey if their opinion isn’t represented or will be annoyed that they weren’t able to be realistic in their answers.

My friend didn’t continue taking the survey.

Three Tips for Effective Interview Guides

by Diane Hagglund

As we’ve said many times before, great research is all about:

Finding the right people to talk to
Asking them the right questions

In our last post we talked about how to use the Recruiting Guide to find the right people to talk to. This post is about the next part, asking the right questions.

Whether an Interview Guide for an in-depth one-on-one conversation or a focus group Moderator’s Guide, the process of writing the document is important. During the cycles of vetting with the client, you gain deep insight into the important subtleties of the findings. There are few hard-and-fast rules for Interview Guides, because the topics vary, but here are a few principles that have worked for us in our research.

1) Invest the time needed to get the participant comfortable – The first question has to be easy and get the participant talking. It has to be more than name and job title or the weather. You need to set the stage for being chatty and opinionated. For our IT participants we like to ask about something happening in the industry, or trace the turns of their career. The catch – this takes time. When the Guide starts getting long and you have to cut, expect someone to suggest eliminating the intro. Don’t. Stick to your guns and spend some time on this. It pays off in a dramatically better conversation and stronger research results when people let their guard down and get chatty.

2) “Peel the Onion” – A good research guide starts more broadly and then gets progressively more specific. For example, start with challenges – what is hard about what they do or what is hard about a particular area. If the challenge the project is trying to address is mentioned unprompted, that is very interesting to know. If it’s not mentioned, go to the next layer of the onion and find out why it wasn’t mentioned. Then drill down one more layer into the actual solution being testing, and so on.

3) Use open-ended questions, but be prepared to prompt – An in-depth interview is not a phone survey. You want to give the participant lots of opportunities to tell you their thoughts on the topic and point to issues that you haven’t thought of. But sometimes people need a gentle push in the right direction, so if your open-ended question isn’t getting you anywhere, be prepared with some options that might get the participant talking. Be sure to present the options in a “let me clarify the question …” tone, not in an “is it A or B” tone.

When Recruiting Guides Go Wrong

by Diane Hagglund

Great research is all about:

Finding the right people to talk to
Asking them the right questions

The Recruiting Guide is the researcher’s tool to ensure we get the first one right – talk to the right people. It is an important written agreement between the researcher and the client that must be approved. This ensures that at the end of the study you know you’ve talked to the right people and helps prevent the “Why did we talk to THAT guy?” syndrome.

But Recruiting Guides can be tricky. The core problem is that prospects don’t talk like marketers do. Marketers like to segment their audience into tidy boxes, but in real life peoples’ roles aren’t that clean. With Recruiting Guides, it is particularly important to be careful about any questions where you ask about responsibility.

IT professionals typically have great pride of ownership in their work – which is great for the businesses that rely on IT business services to function. However, it can be difficult for researchers who need to find a certain type of person, because many people will put up their hands to say they have that responsibility.

In a recent study we were looking for IT procurement participants. We needed to talk to the people who did the hands-on financial and contract work, not the technical buyer who determines the product to purchase and then passes that off to procurement to get the paperwork done. Definitely the way NOT to write the Recruiting Guide in this case was to ask, “Do you have responsibility for IT procurement?” Probably every employee in the IT organization would answer “yes” to that. They all evaluate technology solutions and make recommendations. They feel ownership and responsibility for procurement, even though they don’t have formal procurement roles.

Instead, we asked questions in the guide about reporting structure and the focus of their jobs – how much of their time was spent on procurement? – and we got exactly the people we wanted.

Practical Tip: Think about someone who works with the people you want in your study, but that you don’t want to participate. Imagine how they would answer the questions and pass the screener. It can be challenging, but if you look at every question in the Recruiting Guide and think how someone might qualify for a study even though they aren’t the persona you want, the result will be a better project.

Should Market Research Be Fun for Participants?

by Diane Hagglund

I was reading an interesting market research article by Tim McAtee. This line near the end of the article really jumped out at me: “…this approach can make research more enjoyable for the subject by making survey vehicles less boring.”

This raised a couple of questions for me:

How much fun do our participants have when they are engaged in a project with us?
Does it matter to the outcome of the research?

Fun can certainly add value. Any interviewer knows that getting someone to relax during the interview results in more honest answers and better research findings. We purposely write our interview guides to start off with more general topics so we can get the participant engaged.

Of course, you do have to cover the topic of the research well, but if the participant is qualified to talk about a topic, they probably really enjoy talking about it. And of course a completely boring survey, especially with too many ranking or rating matrix questions, will have a huge drop-off, or if the stipend is really good, they’ll speed through the survey to get the completes without giving thoughtful answers.

On the flip side you can have some bad results in a focus group when there is too much positive energy floating around the room – which is just as bad as an overly negative atmosphere. You can’t have an environment where people are hesitant to share negative viewpoints because they don’t want to bring everyone down.

IMHO there are some definitive statements about fun and market research:

Boring is NEVER good for market research.
Participants who enjoy the process give better feedback.
Fun cannot come at the expense of research goals.

I have one more theory: If the researcher is engaged and enjoying the study, that helps the outcome too. Here at Dimensional Research, we love what we do! We find the IT professionals we have the pleasure to work with endlessly fascinating. They have a truly difficult job and their creativity, tenacity, and smarts consistently impress us.

Technology Market Research: Meet the Real World of Corporate IT

by Diane Hagglund

One of the realities of doing technology market research is that you end up dealing with people in the real world. For those who work in roles that deal only with the hottest new innovations, it can be a bit of a shock to shift gears from the cutting-edge of hype and the super-early-adopters that use new technology.

There is definitely good news. Your technology market research project will give you a good dose of reality and a much better understanding of the market that you’re actually marketing and selling into. And any good market research firm will help you to find exactly the group you need to hear from: whether a cross-section of the entire market, a group of early adopters, or conservative corporate IT executives.

However, bear in mind that the market research project may not feel like the rest of your life. Everyone you talk to on a daily basis may know about your technology and your space, but that doesn’t mean everyone in the world does.

A few important things to remember:

a. “Buzz” usually isn’t happening with the entire market. It may feel like everybody is talking about cloud computing these days, but in reality they aren’t. There are plenty of smart, informed people that simply haven’t got cloud computing on their radar because they are focused on other things.

b. Your competitors are not “everywhere.” We know it feels like that to you, but in reality, only a small percentage of the market uses your competitor’s tools.

c. Even your own customers aren’t as educated about your product as you are. Don’t expect to have the same deep conversation with them that you have in your internal meetings. Remember: you spend 120% of your time thinking about your product. Your customers probably spend only a fraction of their time doing the same.

d. In the real world, corporate IT doesn’t get as excited about change as technology startups do. It may feel like a wet blanket to hear corporate IT research participants finding the negative aspect in the amazing new technology that you know is going to change the world. But the reality is that it’s much better to hear the objections, so you can deal with them.

You should work with your research provider to make sure that you understand exactly who you want to talk to, and it helps to be realistic about the level of effort it takes to find exactly the right people and engage them in a beneficial conversation.

Getting the Feedback that Matters Most

by Diane Hagglund

This blog post can be summed into one sentence: “I talked to some people I know” is NOT market research!

Of course you should talk to people you know about your ideas, but you need to ask yourself, who is your target market?

Will your drinking buddies ever buy your enterprise software? Maybe they will, and count yourself fortunate if you play poker with only CIOs. But you probably interact with a lot of other people who are not actually in your target market.

It’s a good idea to get ideas from everyone, but you should put significantly more emphasis on feedback from the people who are actually part of the community you target.

It may be obvious that your high school buddy who runs his family’s (very successfull) car dealership doesn’t know enough about technology to give you feedback.

The really problematic conversations are usually the ones with people who are in the periphery of your target, just not IN it – people who sell to IT in other companies (especially ones with big established brands!), VCs who invest in tech companies, journalists and bloggers – even your fellow co-workers.

All of these people will have insights for you, and you should certainly pay attention to them, but you should never use the info you get from these people INSTEAD of having conversations with your actual target market, the people who will eventually buy your solution. They are the ones you really need to talk to.

If your day-to-day routine does not easily facilitate those conversations – make a point to make it happen. Find those people and talk to them in a way that gets unbiased feedback. Any market research firm would be happy to help.

Dealing with “Group Think”

by Diane Hagglund

Focus groups are ideal when you want to gain multiple perspectives in an interactive group setting.

However, one of the things that can happen in focus groups is “group think.” The first person starts off on a tangent – good or bad – and then the entire room goes along with that first opinion. This is a dynamic that every good researcher is aware of.

So does groupthink invalidate a focus group? Of course not, and in fact it can be very useful as long as you know it’s happening and stop it when necessary. Here are a few tips:

First, establish if group think is a good or a bad thing for your project. Depending on your goals, group think can be a very bad thing. For example, in message testing you’re usually evaluating the kinds of pain/benefit statements that someone experiences in isolation. Groupthink is very damaging in that kind of environment because you end up seeing only one participant’s uneducated, unguided reaction to messages. In this case, maybe focus groups are not the right medium – in-depth interviews or a Web survey may give you better results.
Remember that group think can be very good. If you’re trying to brainstorm an idea, having somebody to kick off the conversation and then see where the group goes with it is a very good thing. A smart moderator will encourage this kind of thinking for a while and see where it goes. The most creative ideas from focus groups I’ve been in did not come from one single individual. They came from the group members inspiring each other to build off of each other’s experiences and ideas and come up with something more powerful than any of the individual input. When one participants says “what if …”, and then the next participant says “cool idea, but how about this…” and then yet another participant says “or even better, you could take that idea and try…”. Now you’re cooking!
Be prepared to funnel groupthink. When you see it happening, a good moderator will challenge the group with the opposite assumption. Prompt some “group think” around the positive ideas, and then redirect the group to talk about the negative ideas. Insist that the group gives you multiple perspectives.

Above all – don’t be scared of negative group think! It’s very good to hear every possible objection your target audience can make about your solution BEFORE you start selling, so you can be prepared with the products, messages, and objection handling you need to be successful.