Dimensional Research

Technology Market Research

Are Your Research Participants Smart?

by

When doing research, understanding if participants are really experts in their field is important. You have to know if their feedback is coming from a place of knowledge, one of assumption, or perhaps even ignorance.

This can be tricky. There is a type of person who is incredibly good at sounding knowledgeable. They can drop clever sound bites or smiling knowingly to give the impression of deep knowledge, even if they don’t have any.

This is particularly important in our field of IT and business technology market research where many of the concepts are very complex. Even qualified participants who have achieved amazing career success may not know what they’re talking about once you get into lower-level details. This is especially difficult with senior participants who are removed from the actual day-to-day activities, but are hesitant to use those important words “I don’t know”.

Research projects must include perspectives from all kinds of buyers who influence purchasing decisions – which usually includes both “smart” and “not smart” participants. The challenge is two-fold:

  1. Getting a mix of participants that represent the knowledge in your market
  2. Knowing which is which when you present your analysis

Because, here’s the thing. Taking “sounding smart” input and presenting it as “smart” input will not drive business results.

Here’s a few tips for managing this dynamic in research projects:

  • Be “smart” yourself.  You have to have some depth of experience with your topic to know when you’re talking to someone who isn’t an expert. Do your homework prior to any research conversation.
  • Give participants permission to say they don’t know.  Do this directly. Say something like “This is market research – you are not being tested.  It’s us who are being tested. If people like you aren’t aware of a concept that is actually the most important thing for us to learn from this project. Feel free to say you don’t know. That’s a great answer.”  Then praise them when they say  they don’t know.
  • Add a “testing” question.  It may not be the goal of the project, but adding at least one question that’s a bit “in the weeds” and seeing how that is answered will allow you to rate the participant. You should frame it as that though “not to get into the weeds since this isn’t the goal of this project, but I am curious what you think about…”

Market Research: Dealing with No-Shows

by

You have a challenging recruit, but you finally found that last participant who fit the impossible guide that the client has given you. Of course, it turns out that she is available only at 6 AM your time, but you get up and get caffeinated and are ready for a great interview.

You dial. And you get it – the dreaded voicemail.

No problem. People have jobs, they run late, their last meeting hasn’t ended, there was traffic. So you wait five minutes, then try again. And again it is voicemail. You leave a message this time, and then, depending on your best practices, try again in another five minutes or send an email or all of the above.

But at some point you have to admit it – you have a no-show.

We’ve all been there. In B2B market research it’s really a fact of life. Our research participants have jobs that dictate their priorities, which means they sometimes can’t make a call. Since we work with corporate IT, we know that our participants will hang up or even walk out of a focus group if they get an alert that mission critical systems are down. After all, our participants are the heroes who keep their applications up and running so that the business can continue to operate.

So how do you manage no-shows, since the client does require you to talk to a certain number of participants in the project? There are really only two options:

  1. Schedule time for rebooks: Build it into the schedule. If you have ten interviews, a week is plenty of time to talk to ten people. But put two weeks in the schedule anyway so you have time to rebook.
  2. Over-recruit: If the schedule doesn’t allow rebooks, recruit more people than you actually need. This should cover no-shows, and hey, if you get lucky and everybody shows up, the client will be thrilled that you have delivered more input than you were contractually obliged to deliver.

When Recruiting Guides Go Wrong

by

Great research is all about:

  1. Finding the right people to talk to
  2. Asking them the right questions

The Recruiting Guide is the researcher’s tool to ensure we get the first one right – talk to the right people. It is an important written agreement between the researcher and the client that must be approved. This ensures that at the end of the study you know you’ve talked to the right people and helps prevent the “Why did we talk to  THAT guy?” syndrome.

But Recruiting Guides can be tricky. The core problem is that prospects don’t talk like marketers do.  Marketers like to segment their audience into tidy boxes, but in real life peoples’ roles aren’t that clean. With Recruiting Guides, it is particularly important to be careful about any questions where you ask about responsibility.

IT professionals typically have great pride of ownership in their work – which is great for the businesses that rely on IT business services to function. However, it can be difficult for researchers who need to find a certain type of person, because many people will put up their hands to say they have that responsibility.

In a recent study we were looking for IT procurement participants.  We needed to talk to the people who did the hands-on financial and contract work, not the technical buyer who determines the product to purchase and then passes that off to procurement to get the paperwork done. Definitely the way NOT to write the Recruiting Guide in this case was to ask, “Do you have responsibility for IT procurement?” Probably every employee in the IT organization would answer “yes” to that. They all evaluate technology solutions and make recommendations. They feel ownership and responsibility for procurement, even though they don’t have formal procurement roles.

Instead, we asked questions in the guide about reporting structure and the focus of their jobs – how much of their time was spent on procurement? – and we got exactly the people we wanted.

Practical Tip: Think about someone who works with the people you want in your study, but that you don’t want to participate. Imagine how they would answer the questions and pass the screener.  It can be challenging, but if you look at every question in the Recruiting Guide and think how someone might qualify for a study even though they aren’t the persona you want, the result will be a better project.

Rethinking Market Research Incentives

by

Incentives are a normal part of any market research project. With each project we ask ourselves: What will we give participants?  How much will it cost?  How do we sweeten the pot to ensure high participation?

Today we’d like to take a step back and ask:  Do we really need to give people stuff to make them talk to us?

Now, before anybody panics, we do believe that most of the time the answer is “yes.” When we’re reaching out to new audiences that we have absolutely no relationship with, we need something to get their attention.  And if we’re asking for something big – driving across town to attend a focus group for example – of course the stipend is important.  We’ve discussed guidelines for compensating participants in the past, and we still think that post is relevant and useful.

But that doesn’t mean that we always need a “prize.”  Especially when working with customers, the best reward might be that they see the feedback they give influencing the product roadmap or support. Sometimes just being heard is a very strong driver.

For example, we have just completed an annual customer survey for a customer we’ve been working with for years. The first 3 years we gave a typical stipend of a $25 gift card to anyone who completed the survey.   But, this year the budget was slashed, so we figured we’d try it without a stipend and see what happened.  A couple of observations on this test:

  • We got the same number of completes as last year – when we did offer a stipend!  The customer base had grown by 15% so we can expect that the lack of stipend hit our response rate by that much, but we still had a very significant response rate so the results were valid.
  • Responses were higher quality than in the past (as evaluated by the thoughtfulness of responses to open-ended questions).  Since nobody completed the survey just for the $25, participants were more engaged.  They took the time because they cared.
  • Perhaps this wouldn’t have worked the first time? This was the fourth in a series of customer surveys, and this client has definitely responded to the findings of past surveys.  Perhaps their customers now realize the survey is worth their time for other reasons then the stipend?

Net-net: We felt we got at least as good as – and arguably better – results from the customer survey without stipends for less than half the costs of previous surveys. Moving forward, Dimensional Research is making sure we evaluate whether the stipend is truly adding value to a project.

Technology Market Research: Meet the Real World of Corporate IT

by

One of the realities of doing technology market research is that you end up dealing with people in the real world. For those who work in roles that deal only with the hottest new innovations, it can be a bit of a shock to shift gears from the cutting-edge of hype and the  super-early-adopters that use new technology.

There is definitely good news. Your technology market research project will give you a good dose of reality and a much better understanding of the market that you’re actually marketing and selling into. And any good market research firm will help you to find exactly the group you need to hear from: whether a cross-section of the entire market, a group of early adopters, or conservative corporate IT executives.

However, bear in mind that the market research project may not feel like the rest of your life. Everyone you talk to on a daily basis may know about your technology and your space, but that doesn’t mean everyone in the world does.

A few important things to remember:

a. “Buzz” usually isn’t happening with the entire market. It may feel like everybody is talking about cloud computing these days, but in reality they aren’t. There are plenty of smart, informed people that simply haven’t got cloud computing on their radar because they are focused on other things.

b. Your competitors are not “everywhere.” We know it feels like that to you, but in reality, only a small percentage of the market uses your competitor’s tools.

c. Even your own customers aren’t as educated about your product as you are. Don’t expect to have the same deep conversation with them that you have in your internal meetings. Remember:  you spend 120% of your time thinking about your product. Your customers probably spend only a fraction of their time doing the same.

d. In the real world, corporate IT doesn’t get as excited about change as technology startups do. It may feel like a wet blanket to hear corporate IT research participants finding the negative aspect in the amazing new technology that you know is going to change the world.  But the reality is that it’s much better to hear the objections, so you can deal with them.

You should work with your research provider to make sure that you understand exactly who you want to talk to, and it helps to be realistic about the level of effort it takes to find exactly the right people and engage them in a beneficial conversation.

Technology Market Research Should Be Done By Technology Experts

by

I was really surprised to be pointed to this article from Zoomerang (our main online survey tool provider, who I think makes a great survey product). The article states that more than 40% of online IT panels contain survey takers who are not really in IT.

Perhaps I shouldn’t be surprised.

It’s challenging to find appropriate IT participants for research – online or in person.  It requires a very specific knowledge of the people, processes, and technology in IT.  As this article clearly points out, generic market research providers don’t do well here.  I do genuinely think they have TRIED to “fix” this, but I know it’s hard and don’t believe that they really have.

The challenge is that IT professionals are compensated well  for participating in technology market research, so the motivation to misrepresent skills is high.  A better way ensure that you’re talking to real decision makers in corporate IT is to work with a provider who specializes in technology and has years of experience building up the resources to find these people, and has the technical savvy to know immediately if something has gone wrong with the recruit.

Dimensional Research specializes in technology market research, and work extensively with companies that sell to corporate IT.  That’s all we do. And because of that focus, we do it well.  We know the reliable sources to find participants that match your needs.  Just as importantly, we understand technology ourselves,  so we can quickly catch someone who is BS-ing.

Getting the Feedback that Matters Most

by

This blog post can be summed into one sentence: “I talked to some people I know” is NOT market research!

Of course you should talk to people you know about your ideas, but you need to ask yourself, who is your target market?

Will your drinking buddies ever buy your enterprise software?  Maybe they will, and count yourself fortunate if you play poker with only CIOs. But you probably interact with a lot of other people who are not actually in your target market.

It’s a good idea to get ideas from everyone, but you should put significantly more emphasis on feedback from the people who are actually part of the community you target.

It may be obvious that your high school buddy who runs his family’s (very successfull) car dealership doesn’t know enough about technology to give you feedback.

The really problematic conversations are usually the ones with people who are in the periphery of your target, just not IN it – people who sell to IT in other companies (especially ones with big established brands!), VCs who invest in tech companies, journalists and bloggers – even your fellow co-workers.

All of these people will have insights for you, and you should certainly pay attention to them, but you should never use the info you get from these people INSTEAD of having conversations with your actual target market, the people who will eventually buy your solution. They are the ones you really need to talk to.

If your day-to-day routine does not easily facilitate those conversations – make a point to make it happen.  Find those people and talk to them in a way that gets unbiased feedback. Any market research firm would be happy to help.

Technology Professional? Want your opinions to be heard?

by

Are you a technology professional? Do you like sharing your opinions? You might enjoy being part of a Dimensional Research Project.

The only real qualification is that you work with technology in your profession – we’re especially interested in people who work in corporate IT.  We welcome participation from around the globe!

Important information for participants:

  • This is market research – no one will every try to sell you anything.
  • Your input will be anonymous. Nothing you say will ever be attributed to you or your company in any way.

What do you have to do?

Register here (you may need to scroll down a bit). We’ll ask you a few questions about yourself. Then, as research projects come up that you seem to fit, we’ll contact you with some more detailed questions and information about the specific project.

What is involved?

The actual level of participation depends on the project. For our Web surveys, we’ll send you a link and you can go online to complete the survey. For in-depth interviews we’ll contact you and schedule a time to talk – usually about an hour. For focus groups we’ll let you know when (and where, if necessary) and see if you’re available – usually about 1½ – 2 hours.

What’s in it for you?

This depends on the project. For a Web survey, it may be a copy of the final report or a chance to win a cool prize. For a focus group or interview usually it is a cash stipend payable via Paypal or US dollar check. And of course, with all projects you have the knowledge that you’re influencing the future of technology in some way!

REGISTER HERE>>

The Importance of Segmentation in Market Research

by

In my last post, I talked about the similarities across Corporate IT groups when it comes to pain and product benefits.  This week I’d like to talk about key differences that are important to consider when doing research.

This is where segmentation comes into play – making sure you have enough participants of a particular type to represent the differences in your audience. How you segment, and whether you segment at all, depends on your goals. 

For example, if you’re doing a message validation where the goal is to evaluate if you are communicating well about a specific pain or the benefits of your technology, there are a lot of similarities between different geographies and verticals. In this scenario, you don’t need to have a dozen participants from each of the 10 vertical industries you sell into to quickly get a very real sense of how your messages are resonating with a corporate IT audience.

However, in some scenarios, you will need to get representation from different areas. Of course every project is different, and we at Dimensional Research work with our clients to really understand specific goals and make sure we recruit the right representation from any segment to reach those goals. Here are a few general areas to consider. 

The “Silos” of IT

Application Development, Production Operations, Security, …  They all act very differently from each other. They have different cultures:  development likes new features, while “change” often equates to downtime for production operations, and security would ideally prefer that nobody gets access to anything.  They purchase differently.  And (let’s be honest) these frequently have challenges communicating with each other.  If you have a product that crosses IT silos, you need representation from each silo to understand purchasing behaviors, objections, etc. 

Technical Buyer vs. Economic Buyer

The technical buyer will discuss features and functions.  This is a comfort area for many technology companies and they’re more likely to “get” you and your product.  However, the economic buyer is vital to tell you the non-product reasons why people buy – brand, sales execution, channel, financing, etc.  You will need both types of participants to understand competitive dynamics and purchasing behaviors.

Verticals

Government, Education and Non-profits can purchase very differently than IT in other vertical industries.  If you are targeting those verticals, you need to talk to them specifically about how they buy. 

Financial Services spends more on IT and is most likely to consider a new technology for competitive advantage.

ISVs and SAAS are organized and make purchase decisions much differently than corporate IT – although they often are an important market for technology vendors.

Geographies

There are big differences in how these geographies actually purchase products.  Culture can play a big part in acquiring a technology.  European countries frequently prefer working with local representatives – regional offices or channel partners.  Japan absolutely requires local language and culture in their purchasing process.  Canada acts like an extension of the US in the way they purchase and can be easily combined with a general North American study.

Technology and Process Maturity

If you are selling a cutting-edge technology, the best places to do market research are the San Francisco Bay Area, New York, London, and Sydney.  These are all cities with cultures of trying new things. 

On the other hand, if you want ITIL process maturity, Europe is great. And I absolutely love the mid-west (Chicago, Minneapolis, etc.) for identifying “objections” in a very straightforward and honest way 

Bottom line: You should work with your market researcher to figure out your goals and to determine which segments of the market you need to talk to. Then make sure you get the representation from these segments when executing your market research project.

Six guidelines for compensating research participants

by

There is usually some form of compensation for participating in a market research project. For focus groups and in-depth interviews, research participants are usually offered some kind of stipend in appreciation of their time.  The question I get asked frequently is – how much?

I’ll start by saying that the stipend is usually NOT the primary motivation for participating in technology market research.  I find people are genuinely interested in expressing their opinion. They like to be heard; they like to hear about new ideas coming down the pipe; they like thinking they are influencing the market or the product; and – in the case of focus groups and customer advisory boards – they like to hear what is happening with their peers.  And let’s face it, corporate IT employees are typically paid reasonably well, so a hundred bucks isn’t going to really compensate them for trucking across town to a focus group facility, spending 2 hours talking to you, then trucking back home.

That said, the stipend is key in attracting the right audience, and is perceived as an important added benefit. So back to the question of how much.  The answer depends on a variety of factors:

1. What are you asking people to do?  Focus groups typically take about 2 hours of participants’ time. In-depth interviews usually take from thirty minutes to one hour. In addition, for in-person focus groups, the participants need to get to a specific location.  As a result, focus group participants are usually compensated more generously.

2. How big is the target pool of participants?  The harder they are to find, the more you should sweeten the pot to simplify the recruiting.  If you have a straightforward recruit like “application developers” or “network administrators”, there are plenty of those and you don’t need to have a particularly large stipend to entice them to join. 

However, if you need something really specific like “DBAs responsible for MATISSE databases”, or customers of a competitor that only has a few hundred users worldwide, you want to have a large stipend to make sure that you’re doing everything possible to attract the few people out there that match the recruiting profile.

3. What level of participants are you looking for?  CIOs and other IT executives make more money than sys admins or developers, so the stipends need to be higher to match their expectations for the value of their time.

4. Customers or prospects?  Depending on the context, offering a customer money to give you feedback may be tacky.  Particularly if it’s a customer advisory board meeting, the real value is the chance to be heard.  In this case a nice gift with the corporate logo on it is much more appropriate.  

5.  Give the philathropic option.  More and more corporations have implemented strict policies that employees CANNOT receive any gifts of any kind.  Dimensional Research always offers our research participants the option of donating their stipends to the charity of their choice. 

6.  Web surveys are different.  When you’re looking for hundreds or even thousands of respondents, any stipend, no matter how small, can quickly blow up your research budget. There are certainly options like being entered into a drawing for a gift certificate.  Or the ever popular drawing for the hot electronic item of the day – the iPod Touch has been particularly common in survey drawings in the past year. 

One thing to consider, if the findings are not confidential, is to simply offer a copy of the final report to participants.  This is a very high-value offer to people who care about the topic – probably even more than a gadget – and has the added benefit of not attracting participants who don’t care about the topic and might not give the most insightful or informed answers.