Dimensional Research

Technology Market Research

ROI Analysis

by

It’s a bit of a Catch-22 for vendors who sell to Corporate IT. In pre-sales situations, customers demand to see evidence of ROI achieved by other customers. They want calculations and solid numbers. But then when they deploy your product they don’t track their own ROI, so you can’t share that with new prospects.

For what it’s worth, Corporate IT knows that they do this and that it isn’t fair.  I’ve talked about this with many participants in interviews and focus groups and they understand the dynamic very well.  They do appreciate your efforts to get them anything, even if it’s clearly an internal ROI that is heavily biased. But of course, they really do want that real ROI since their management is demanding it from them.

In response to direct client feedback, Dimensional Research has launched a new offering for establishing ROI, The ROI Framework. In this project, we interview your customers to get their ROI – whatever they have. Some will be solid, some will be anecdotal, but when you put together the information from 5-10 or more of these customers, it’s pretty compelling. To make it easy for customers we’ll offer anonymity so they can be completely open about their ROI and not have to jump through hoops getting corporate approvals.

We’ll take those interviews, and put together a white paper summarizing the input that can be used as a collateral piece. We will also use the content to develop an ROI calculator for use as a sales tool or on your web site. Now your ROI calculator will have as a source “Dimensional Research Customer Analysis” rather than “our best internal guess.”

Contact us today if you’re interested in helping your sales force beat the Catch-22 of ROI.

Technology Market Research: Best of 2009

by

As New Year’s approaches, anyone surfing the Internet knows that it becomes increasingly hard to avoid “best of” lists. We went back and forth on whether to publish such a list, then decided that since blogs are arranged in a chronological order, highlighting the “recent” rather than the “best,” it’s sometimes difficult for readers to find the best material in a blog. This list should help – it highlights our favorite 2009 market research articles, as well as reader favorites.

Best of 2009: Reader Favorites

  • In-Depth Interviews, Focus Groups, or Both? When doing qualitative research, we need to decide which is right for the client: in-depth interviews, focus groups, or maybe a combination of both. The choice depends entirely on the client’s goals.
  • Top Three Questions about Competitive Research. As technology market research experts, we get a lot of inquiries about competitive research projects. This article includes answers to some of our most frequently asked questions.
  • Customer Satisfaction Surveys: Avoiding “Survivor Bias”. One of the biggest mistakes in doing customer satisfaction surveys is allowing “survivor bias”.  This happens when you surveying your existing “surviving” customers – the ones that stayed with you no matter what – and ignoring the customers and business that you lost.

Best of 2009: Our Favorites

We know that these articles are relevant, because we keep sending them to clients! Among other issues, these articles answer the important question of “what is technology market research?” and explain how to measure market research ROI.

  • Technology Expert Defined. Technology market experts are experts on Corporate IT and how it works. They are familiar with Corporate IT and its processes and can talk comfortably and knowledgeably about technology, processes and people in the IT industry.
  • Message Validation: Market Research with Clear ROI. Hard ROI is easy to demonstrate when your marketing programs get two times, ten times or even greater response from the same spend, because your message is more compelling.
  • Involve Your Customers In Market Research. Customers love being involved in market research. They don’t see it as a chore or just another thing on the “to do” list. On the contrary, they love being involved and appreciate the opportunity to voice their opinions.

These are our best, and most useful, technology market research articles for 2009. There are many more articles on this blog (over 50 articles in fact), so do feel free to browse. Some of our favorite categories are Quantitative or Qualitative and Research Questions.

Happy New Year!

Web Surveys vs. Phone Surveys

by

One approach to market research that we haven’t talked much about in this blog is phone surveys.

Phone surveys are when a person with a nice voice calls people and asks questions from a script. Answers are recorded in a spreadsheet, and the final result is very similar to the graphs you’d expect out of a Web survey.

Phone surveys are different from in-depth interviews because the script is asked exactly as written, unlike an in-depth interview where you have a researcher who is a technology expert asking the follow-on questions needed to drill down into answers.

Dimensional Research usually doesn’t recommend phone surveys to our technology clients. Even for consumer marketing, phone surveys are becoming less useful according to Jay Leve of SurveyUSA: “There is [no] future for any form of telephone research that is predicated on the researcher being able to barge in at will and seize the respondent.”

Several more compelling reasons why phone surveys don’t work are outlined very nicely by Jeffrey Henning at Research Live. The ones that relate to market research for technology companies include:

– Expense of dialing. More and more phone surveys are done via cell phones, since more and more people (and this includes many technology startups) don’t use landlines. In the US, by law, you can only use automatic dialing for landlines, not for cellphones. Manual dialing is much more labor-intensive – and expensive.

– Online surveys eliminate the expense of data entry. The respondent to a web survey is, in effect, donating the data entry cost, as they select the appropriate choices and type in their answers. With a phone survey, you are paying a call center representative to transcribe each respondent’s replies.

– The visual medium of Web surveys lets you easily show people visual concepts, such as ads or core messages, and get their response. Surveys that require respondents to react to visual concepts can’t be conducted with phone surveys alone.

– Web surveys have the allure of confidentiality. People today feel more comfortable sharing information on the Web than answering the prying questions of a phone interviewer.

– People prefer Web surveys, because a Web survey can be done at the respondent’s convenience, rather than at the moment the phone interviewer happened to call.

Jeffrey does make a case for phone surveys in certain situations, but almost none of these are relevant to IT market research including:

  • Major Account Research – In these cases we recommend in-depth interviews or online customer advisory boards. Why wouldn’t you take the opportunity to have a deep conversation with your biggest customers?
  • The Human Touch – This argument only works with Corporate IT if you are also knowledgeable about technology, so again, it’s better to do in-depth interviews.
  • Some People Aren’t Online – This is obviously not an issue for technology professionals.

Our recommendation, after years of doing market research with Corporate IT, is to avoid phone surveys when doing technology market research. Instead, use Web surveys, in-depth interviews, or a combination of both.

Bad Survey Design – Ouch!

by

Whenever we get a chance, we love to participate in market research done by other companies. Doing so gives us the opportunity to think about how we’re answering questions about the stuff we care about. Being on the “other side of the glass” is a great reality check.

We recently upgraded some important business software that we use here at Dimensional Research, and had the opportunity to do a follow-up survey on the experience.  We quickly discovered that it was a very poorly designed survey. The company wasted their time and energy (and ours too!) on a survey that was so badly designed, it will do nothing to help make things better next time.

Here are the things they did wrong:

1) To get the product installed, you first had to register, get a new license key, download the product, then run the installer. Four distinct steps – and we had problems with the first three! But the survey asked only about the step that actually worked – the last one. There was no opportunity to tell them that we had to register three times, that we ended up phoning somebody to get our license key, and that the first site we were sent to for the download didn’t have the right file.

2) They were clearly using the same exact survey for people who were first-time buyers and for those who were upgrading. Even though we purchased three years ago and our reasons for purchasing aren’t relevant anymore, we were required to answer questions about how we chose this vendor, etc.  Even if their lists are so bad that they don’t know we were an upgrade rather than a new purchase, a better designed survey would have given us the opportunity to skip new purchaser questions and continue onto the relevant product-related questions.

3) They did not ask any product-related questions aside from the installation. We’ve been using this product for three years now and they’ve never asked us any questions about the product – only about purchasing and installing the product. It definitely left us with the impression that all they care about is getting our renewal dollars .

4) There were no open ended questions where we could inform them of our issues, so that they could fix them, or to find out if we were an isolated case or if all of their customers have had the same problems.

Today’s fabulous, easy-to-use online survey tools are enabling a lot of bad survey behavior. We trust the readers of this blog will have better judgment!

Technology Market Research: Meet the Real World of Corporate IT

by

One of the realities of doing technology market research is that you end up dealing with people in the real world. For those who work in roles that deal only with the hottest new innovations, it can be a bit of a shock to shift gears from the cutting-edge of hype and the  super-early-adopters that use new technology.

There is definitely good news. Your technology market research project will give you a good dose of reality and a much better understanding of the market that you’re actually marketing and selling into. And any good market research firm will help you to find exactly the group you need to hear from: whether a cross-section of the entire market, a group of early adopters, or conservative corporate IT executives.

However, bear in mind that the market research project may not feel like the rest of your life. Everyone you talk to on a daily basis may know about your technology and your space, but that doesn’t mean everyone in the world does.

A few important things to remember:

a. “Buzz” usually isn’t happening with the entire market. It may feel like everybody is talking about cloud computing these days, but in reality they aren’t. There are plenty of smart, informed people that simply haven’t got cloud computing on their radar because they are focused on other things.

b. Your competitors are not “everywhere.” We know it feels like that to you, but in reality, only a small percentage of the market uses your competitor’s tools.

c. Even your own customers aren’t as educated about your product as you are. Don’t expect to have the same deep conversation with them that you have in your internal meetings. Remember:  you spend 120% of your time thinking about your product. Your customers probably spend only a fraction of their time doing the same.

d. In the real world, corporate IT doesn’t get as excited about change as technology startups do. It may feel like a wet blanket to hear corporate IT research participants finding the negative aspect in the amazing new technology that you know is going to change the world.  But the reality is that it’s much better to hear the objections, so you can deal with them.

You should work with your research provider to make sure that you understand exactly who you want to talk to, and it helps to be realistic about the level of effort it takes to find exactly the right people and engage them in a beneficial conversation.

Market Research: Guiding Responses

by

In this blog, we’ve talked a lot about market research question design, and for a good reason. When preparing a market research project, it’s important to understand how you’re leading your audience. Of course you need to guide discussions and probe for specific responses, but you need to be very aware of what you’re doing and how the way you ask influences the responses.

A very simple example of a subtle guidance: When taking personality tests, if you say “take your time” you get different responses than if you say “do your best”. “Take your time” implies that you want very thoughtful answers, and people respond accordingly. “Do your best” is neutral and doesn’t have  implicit guidance built in.

Think about your research goals, and set up your questions appropriately.

If you’re doing a message test where in real life people don’t take a lot of time to think through underlying ideas, create a similar environment with guidance like “Please review and give me your first gut reaction” or “Don’t over-think this, I’m looking for your initial response.”

If you’re looking to understand pain, it may take some digging to get to the source because your participant may not actually be aware of what’s going on.  This is particularly true with technology professionals that have lived with the pain for a while and have a workaround in place. They may not notice how much time they spend doing something that is not value added because it’s just the way it works.  In that case, ask questions that encourage  thoughtfulness, such as “Why is that?” or “Really, tell me more about that” or “Could a different approach be more valuable?“

The Top Three Questions about Competitive Research

by

One of the great uses of market research in general, and technology market research in particular, is getting G2 on what your competitors are doing.

Market research organizations usually have much greater access to your competitors’ customers, and the anonymity of a formal market research process allows participants to be much more open.

As technology market research experts, we get a lot of inquiries about competitive research projects. Here are answers to three of our most frequently asked questions:

1. Can you find my competitor’s customers?

Yes, with only one caveat – they actually have customers!!! Getting participants for any market research project is a bit of a numbers game. There will always be some percentage of people who don’t have the time or simply aren’t interested in participating. If your competition only has 20 customers in the entire world, we probably will struggle to find them all and get them to agree to participate in a project.

But with this one caveat, we have tried-and-true methods for finding users of any particular technology. In fact, at Dimensional Research we have actually had clients with such bad internal data and complex internal processes, that it’s been easier for us to find their customers!

2. Will my competitor’s customers talk to you?

Yes, absolutely. Not only do we motivate them with appropriate compensation but people like to be heard. And technology professionals know that strong competition drives innovation, so they want their vendors to have competition, and they want to have options.

3. How specific will competitor customers get with their info?

That of course depends on the participant and how open they are, but the majority of participants will tell you everything they know. It depends to some degree on the goals of the project. If your goals are to understand the motivation for purchase – such as a win/loss project or a lost-deal analysis – those are very straightforward questions and it’s straightforward to get clear answers from customers.

If your goal is to understand weaknesses in your competitor’s product, that can also be done, but you do need to be prepared that existing customers have often worked through weaknesses, have figured out workarounds, and no longer perceive those as problems. Having some sense of what the weaknesses might be so you can encourage the participants to recall their initial response to those problems will give better results.

Remember though that this is a good news/bad news situation. It’s just as easy for YOUR competitors to talk to your customers, so at Dimensional Research we strongly recommend that you get feedback from your own users in addition to your competitors’ customers!

Defining Terms in Market Research and Beyond

by

The other day I was talking to a client about cloud computing, and he was frustrated. He finally said, “Does ‘cloud’ stand for something in the technology, or does it stand for the lack of clarity around the concept?”

His response made me laugh, but it also reminded me about how important it is to be very clear about terms when you want people to understand you.

When I was on my high school debate team, and again later on when I studied mathematics, you always had to start by defining your terms. It was a given that you couldn’t have a discussion or prove anything unless you knew what you were talking about in the beginning.

Talking with corporate IT you have to do the same thing – especially with new technology and acronyms! When you’re talking about PAAS, SAAS,  cloud, application virtualization , ITSM, or any other jargon, be kind and give the audience a clue about what exactly you’re talking about. Never assume they already know.

And when doing technology market research, don’t guess!!! Either tell the participants exactly what you mean, or ask questions about what the participants think it means. Put in the effort to make sure you understand each other.

Customer Satisfaction Surveys: Avoid “Survivor Bias”

by

One of the biggest mistakes in doing customer satisfaction surveys is allowing “survivor bias”.  This happens when you surveying your existing “surviving” customers – the ones that stayed with you no matter what – and ignoring the customers and business that you lost.

“Survivor bias” refers to drawing conclusions only from data that is available or convenient and thus systematically biasing your results. In technology market research, it means neglecting former customers and only drawing conclusions from existing customers – which are by definition happy customers (or at least happy enough that they have not stopped doing business with you).

If you really want to understand customer satisfaction, don’t just get feedback from the people who continue to be your customers no matter what you do to them. You need to include your former customers – those who have turned to your competitor or just plain don’t use the solution anymore.

Allowing survivor bias is a particularly heinous research mistake because it’s not hard to avoid.   The data is there after all – if you have data on your customers, you have data on your former customers.

This raises one very important question – do you even know that they are “former?”  The research can help you find that out, and even more importantly, find out why they are no longer customers. If there’s a problem with your solution, you need to know about it, and the sooner the better.

In this year’s annual customer satisfaction survey, make an effort to include former customers. After all, as we said here before, your goal is to get honest feedback, and when it comes to market research, negative feedback is good for you!

Technology Market Research Should Be Done By Technology Experts

by

I was really surprised to be pointed to this article from Zoomerang (our main online survey tool provider, who I think makes a great survey product). The article states that more than 40% of online IT panels contain survey takers who are not really in IT.

Perhaps I shouldn’t be surprised.

It’s challenging to find appropriate IT participants for research – online or in person.  It requires a very specific knowledge of the people, processes, and technology in IT.  As this article clearly points out, generic market research providers don’t do well here.  I do genuinely think they have TRIED to “fix” this, but I know it’s hard and don’t believe that they really have.

The challenge is that IT professionals are compensated well  for participating in technology market research, so the motivation to misrepresent skills is high.  A better way ensure that you’re talking to real decision makers in corporate IT is to work with a provider who specializes in technology and has years of experience building up the resources to find these people, and has the technical savvy to know immediately if something has gone wrong with the recruit.

Dimensional Research specializes in technology market research, and work extensively with companies that sell to corporate IT.  That’s all we do. And because of that focus, we do it well.  We know the reliable sources to find participants that match your needs.  Just as importantly, we understand technology ourselves,  so we can quickly catch someone who is BS-ing.