Dimensional Research

Technology Market Research

Always Ask the Question

by

Methodology matters. Last night I had one of those moments that reminded me again that a structured research interview is not just a conversation.

I was on the phone with an IT manager talking about a product he had used for about a year. His experiences were bad. He had harsh things to say about the product, the implementation, and the lack of business results. He had been  articulate and forceful about his frustrations, so when I got to the “would you ever recommend or purchase this product again” question I was confident the answer would be a resounding “NO! NEVER! NOT EVER!”.

I was wrong. In fact, my participant had changed jobs in the past two weeks and he had already spoken to his new management about purchasing the product – the exact same terrible one we’d been discussing. I was really surprised and asked again to make sure I’d understood. It turned out that while he didn’t like the product and thought it was terrible, it was still much better than nothing. But I needed to ask the question to figure that out.

In a social conversation, it’s ok to project and draw conclusions. You end up with a more interesting exchange. But in a research interview, you can’t. It’s too important to understand all angles.  So always ask the question, even if you are confident you know the answer. 

Are Your Research Participants Smart?

by

When doing research, understanding if participants are really experts in their field is important. You have to know if their feedback is coming from a place of knowledge, one of assumption, or perhaps even ignorance.

This can be tricky. There is a type of person who is incredibly good at sounding knowledgeable. They can drop clever sound bites or smiling knowingly to give the impression of deep knowledge, even if they don’t have any.

This is particularly important in our field of IT and business technology market research where many of the concepts are very complex. Even qualified participants who have achieved amazing career success may not know what they’re talking about once you get into lower-level details. This is especially difficult with senior participants who are removed from the actual day-to-day activities, but are hesitant to use those important words “I don’t know”.

Research projects must include perspectives from all kinds of buyers who influence purchasing decisions – which usually includes both “smart” and “not smart” participants. The challenge is two-fold:

  1. Getting a mix of participants that represent the knowledge in your market
  2. Knowing which is which when you present your analysis

Because, here’s the thing. Taking “sounding smart” input and presenting it as “smart” input will not drive business results.

Here’s a few tips for managing this dynamic in research projects:

  • Be “smart” yourself.  You have to have some depth of experience with your topic to know when you’re talking to someone who isn’t an expert. Do your homework prior to any research conversation.
  • Give participants permission to say they don’t know.  Do this directly. Say something like “This is market research – you are not being tested.  It’s us who are being tested. If people like you aren’t aware of a concept that is actually the most important thing for us to learn from this project. Feel free to say you don’t know. That’s a great answer.”  Then praise them when they say  they don’t know.
  • Add a “testing” question.  It may not be the goal of the project, but adding at least one question that’s a bit “in the weeds” and seeing how that is answered will allow you to rate the participant. You should frame it as that though “not to get into the weeds since this isn’t the goal of this project, but I am curious what you think about…”

Tips for Doing Qualitative Global Research

by

I was asked the other day about how to choose a research firm for qualitative global research projects – interviews and focus groups.

NOTE:  We work exclusively with technology, so this advice may not translate to consumer products – what do you think? Do let us know in the comments.

At Dimensional Research, we’ve recently done a number of projects in India, China, Japan, Australia, and Singapore, as well as Germany, France, UK, and other countries in Europe. We’ve also done virtual projects with participants from Peru, Brazil, Ecuador and Venezuela.

We have found that the principles are basically the same in any international country, with special caveats where English is not the first language.

Our recommendation is to prioritize deep experience in the industry rather than first looking for deep experience in a particular country. It is always possible to find local language support, but almost impossible to find the combination of both language and technical expertise. Of course, a firm has to have some sense of doing international research and know the pitfalls, but once you’ve done it in a few countries, the process translates to almost any other country.

The most typical approach is to pick the locations, and then partner with a local facility. Since recruiting works straight off the guide, people answer the questions the way the guide is written it to determine qualification. The important step is getting a really good guide that does a great job of filtering out unqualified people – if you do this, the local people can easily translate and get a good recruit.

Firms like Dimensional Research keep track of the really great research facilities in various countries through our direct experience and our research network – and since we specialize in technology, we work only with facilities that do great technology work. That evolves over time since there is a lot of consolidation in the market research facilities business, so it’s an ongoing effort.

In terms of moderation, if it is at all possible, we conduct research in English. Since English is the language of corporate IT and business technology, that is a real option in our research. While you need to give participants more time to communicate, the results are better if English is an option.

But it’s not always possible – in Japan it’s nearly impossible to do English-language research. In China, it depends on the audience – if you deal with very senior IT executives in the modern cities of China you might be able to use English, otherwise probably not. We have found some really excellent Japanese and Chinese moderators that we use on our projects who know technology really well and we coach them heavily before going into projects.

Bonus Tip: Make sure any firm you work with is aware of and embraces international culture. It really does make a difference to have experience outside the US to pull from. A big part of a researcher’s job is to put the participants at ease so they want to tell you the stuff that is a bit sensitive or embarrassing or doesn’t make them or their company look good.

During one of my first experiences with international research (I was only an observer), the moderator was an incredibly nice man who clearly believed that the US was the best country on earth. Unfortunately he didn’t quite get it that the British people he was talking to didn’t seem to agree with him. Not the best place to start a research conversation!

Finding the Story in the Research

by

On a big reading “catch up” over the long weekend, I read two blogs posts that caught my attention:

  1. The first was Seth Godin’s Are you doing math or arithmetic? This blog post resonated because my degree is in mathematics. [Side comment:  The best way to horrify your math professor is to refer to basic arithmetic activities like addition or fractions as “mathematics.” Mathematics has nothing to do with computation at a certain level. It’s about taking your facts (called axioms), and applying pure logic to come up with a proof. Knowing that something is unarguably true or false – that is the real beauty of mathematics!]
  2. I also read Ivana Taylor’s challenging blog post about the future of market research firms. Ivana beautifully outlined the decreasing value that exists in collecting information – it’s simply too easy to do that these days.

So the question:  where is the value – or  the “mathematics” as Seth Godin would say – that clients pay research firms good money for.

The massive report that shows everything is not it. It’s not that difficult to spew the data from the information gathering phase into a document. It does provide value to the internal people who take the time to read and absorb the details and is not a step that should be skipped. But let’s face it, that part is the “arithmetic.”

There is a point in every research project where you have to take those massive amounts of information, gathered over a huge number of interactions, and figure out how to communicate what matters in a one-hour, or a 30- minute or even a 10-minute meeting.

This is where the value is found – in using the information to build a compelling story. This is the “mathematics” of research. Creating the story of reality  in a way that is understood and actionable by a wide audience.

Very easy to say. Very, very difficult to do. Extremely high value when you do it.

Three Tips for Effective Win/Loss

by

It seems that win/loss analysis is in the air these days. We’ve had a sharp uptick in incoming inquiries for projects analyzing why enterprise software deals are won or lost when there is a competitor in the mix.

The major factor in successful Win/Loss projects is the ability to identify stakeholders who were deeply involved in the process and are willing to talk about their experiences. Here are our top tips to get the right people:

  1. Ensure anonymity – during an enterprise sales process, prospects can develop relationships with their account management team. They may be hesitant to discuss issues that happened during the sales process if they are worried that they’ll be putting someone they know in a bad light.
  2. Expect more effort to secure losses – it’s easier to get people to talk about wins. Once a company has chosen a vendor to have a relationship with, they are more vested in that relationship and willing to put the time into making it work. If a customer has chosen another vendor, they will be less likely to talk to you. Plan for this when building target lists.
  3. Keep it short – Senior IT executives value their time above all else. They are busy. It’s difficult to get an hour of their time, but asking for 15 or 30 minutes feels like much less of a commitment and they will be more likely to participate than if they feel it is a big and onerous conversation. Of course, once they’re on the phone, if they want to keep talking let them! But don’t start with the big ask.

Bad Surveys: Participants Know When Research is Bad!

by

A friend of mine who is not involved in the research business sent me this email yesterday.

Hi Diane.

I attended a conference this week and was sent this survey to fill out.

Did you watch the playback of the keynote session?
( )Yes. It was excellent.
( )Yes.  It was OK.
( )No, but I plan to.
( )No, and I am not interested.

What I want to know is where is the response “Yes and it sucked!”

Of course this was a bad survey question, which we’ve talked about before  (Bad Survey Design; Bad Web Survey Question).

But the broader point isn’t just that the research is bad, but that THE PARTICIPANTS KNOW! They can tell if their option isn’t on the list and they just won’t complete the survey if their opinion isn’t represented or will be annoyed that they weren’t able to be realistic in their answers.

My friend didn’t continue taking the survey.

Market Research: Dealing with No-Shows

by

You have a challenging recruit, but you finally found that last participant who fit the impossible guide that the client has given you. Of course, it turns out that she is available only at 6 AM your time, but you get up and get caffeinated and are ready for a great interview.

You dial. And you get it – the dreaded voicemail.

No problem. People have jobs, they run late, their last meeting hasn’t ended, there was traffic. So you wait five minutes, then try again. And again it is voicemail. You leave a message this time, and then, depending on your best practices, try again in another five minutes or send an email or all of the above.

But at some point you have to admit it – you have a no-show.

We’ve all been there. In B2B market research it’s really a fact of life. Our research participants have jobs that dictate their priorities, which means they sometimes can’t make a call. Since we work with corporate IT, we know that our participants will hang up or even walk out of a focus group if they get an alert that mission critical systems are down. After all, our participants are the heroes who keep their applications up and running so that the business can continue to operate.

So how do you manage no-shows, since the client does require you to talk to a certain number of participants in the project? There are really only two options:

  1. Schedule time for rebooks: Build it into the schedule. If you have ten interviews, a week is plenty of time to talk to ten people. But put two weeks in the schedule anyway so you have time to rebook.
  2. Over-recruit: If the schedule doesn’t allow rebooks, recruit more people than you actually need. This should cover no-shows, and hey, if you get lucky and everybody shows up, the client will be thrilled that you have delivered more input than you were contractually obliged to deliver.

B2B Market Research: Special Considerations

by

I was delighted to participate in last week’s Market Research Twitterversity. The topic was Project Management, and I was a “guest professor” for the session on Special Considerations for B2B Research Projects.

B2B Market Research: How Is It Different?

The first step in preparing for this session was defining what made B2B market research projects different than those focusing on consumers. Many factors are very similar, but there are two important things that stand out as very unique:

  1. B2B purchases are complex: Businesses have a lot of stakeholders who impact purchases – economic buyers who own the budget, end users who drive requirements, procurement, legal, and more. Decisions are often global, which adds cultural, language and timezone complexity. Understanding these behaviors and incorporating them into the project is vital to ensuring that research goals are met.
  2. Participants have jobs: The reason why you want to talk to participants in B2B projects is that they work for companies. But that means your potential participant pool is busy, has other priorities, and is frequently very well paid. It’s harder to get their attention.

Managing Schedule Risk

When talking project management, a typical question is “How long do these projects take?” We’ve blogged about this topic before, and have given general guidelines. However, the key to meeting any schedule is identifying and managing timeline risks. At Twitterversity we discussed schedule risk specific to B2B projects, and gave some tips for handling it, including:

  • Plan for no-shows: Participants have jobs and those always take priority. No shows are a fact of life in B2B research. Either put time in your schedule for rebooks, or over-recruit to meet project goals.
  • Be realistic about customer information: Companies often have terrible information about their own customers, or there is an account manager that must give permission before you can make a call, or there are strict usage guidelines for communicating via email. Any of these things can derail a project schedule. If your project depends on customer information, have a “straight-face” conversation early on with the gatekeeper of that data to find out the real status of that information and your ability to access it.
  • Understand confidentiality issues: You must find out if participants are unable or unwilling to talk about their business in front of their competitors. And this has to happen BEFORE they are all sitting together in a focus group. If you are aware of issues early, you can build the project to ensure that no participants from competitive companies cross paths.
  • Capture international needs: Regulations, language and cultural differences are often important findings in a research project, but discovering those adds complexity to the project and must be managed.
  • Understand verticals: Is there only one vertical that matters? How are those weighted? In B2B projects it can be particularly important to understand the importance of government, non-profit, and education since those three verticals purchase very differently. Do you include, exclude, or perhaps even over-represent those?

In spite of these extra considerations, B2B projects can be very rewarding. We are always learning something new!

Bonus Tip

For any research project – not just B2B – stakeholder management is important. We recommend Patty Azarello’s great blog. It is not research-specific, but it gives some great insights into how to manage your stakeholders.

Market Research: The Importance of Vocabulary

by

Model Metrics, a provider of services at the cutting edge of cloud computing, recently sponsored a Dimensional Research survey on iPad and tablet adoption in the Enterprise. It was a good project with a fun topic and the chance to work with great client. It also demonstrated very clearly the importance of knowing how your audience uses vocabulary.

Consider the following two findings from the survey:

  • 22% of IT professionals reported having officially deployed tablets
  • 72% have tablets in use for business purposes in the enterprise

If you don’t know the language of corporate IT, this probably doesn’t make sense. How can there be iPads in use for business purposes when they haven’t been deployed yet?

The answer is in the word “officially.” One of the most interesting findings of the study is that employees are purchasing iPads or other tablets on their own, then bringing them to work, and IT has to support them. IT hasn’t deployed the iPads “officially,” but they have to support them anyway.

A great example of how words matter in market research. You can read more about the Model Metrics study here, or download the full report here.

Research Bias: It Can Happen to Anyone

by

We are nearing the end of our year living in Europe.  One of our goals for this trip was to experience the rich Roman and Medieval history in the area – most of which requires driving south. It is very quick to get around France once you are on the freeway, but freeway entrances are fairly far apart.

From our house, you can get on the freeway by driving north or south. The south entrance, we will call this Route A, is further away, but is in the right direction once you are on the freeway. Here is the Google Map from close to our house to the town of Orange.

The north freeway entrance, we will call this Route B, is closer, but is north from where we live so takes you out of the way. However, you spend more of the time driving freeway speeds.  Here it is on Google Maps.

Now here’s the thing – my husband and I do NOT agree on which route to take, and driving in the car offers plenty of time to discuss why our particular point of view is the correct one.

My husband started with the stats from Google Maps to support his point of view:

  • Route A: 46.3 km,  47 minutes  (shorter and faster)
  • Route B: 64.3 km, 51 minutes (longer and slower)

I pointed out that Google Maps is based on algorithms and not reality, so suggested we do some real-life measurements. The next few trips south I wrote down our departure times and arrival times at a set point on the freeway both coming and going (yes, we are that kind of couple!) using both routes:

  • Route A: 46 minutes, 58 minutes, 53 minutes, 52 minutes  (faster 1 out of 4 times)
  • Route B: 50 minutes, 52 minutes, 51 minutes, 49 minutes  (faster 3 out of 4 times)

Now, it should be immediately obvious that my husband is an advocate of Route A, and I prefer Route B. And we both have data to prove that we are right. My husband would point out that my measurements were invalid since on the trips where Route A was slower we ran into unusual traffic. I would point out that if we ran into unusual traffic 3 out of 4 times it probably means that the traffic is not that unusual.

Of course, we should know better.  He is an engineer and I am a mathematician. Before we started all our “fact finding,” we should have just put on the table that we are biased. I don’t like roundabouts, so getting on the freeway sooner is a way to avoid them.  My husband finds the freeway boring and prefers going through the towns.

So back to the market research topic. In this case the final result doesn’t really matter. But in any market research project, it’s important to know and confront biases at the beginning of a project, so you can ensure they are addressed. Findings must be strong enough that exceptions that support existing biases are identified and understood.