Dimensional Research

Technology Market Research

Three Tips for Crafting Better Online Surveys

by

There are many ways to make sure your online survey is efficient and effective. One of them, of course, is to avoid asking bad questions.  

Tip 1: Craft your questions carefully to avoid unwanted results 

This tip was inspire d by Seth Godin. He explains that “Every question you ask changes the way your users think. If you ask, ‘which did you hate more…’ then you’ve planted a seed.” 

Mr. Godin makes a great point. I recently booked a trip with Travelocity. It was a great trip. I was happy.  I did have a small issue with the airport transfer getting home, so I filed a complaint to see if I could get a refund.  It was a small matter – about $30 credit – so I wasn’t too worried about it. Travelocity’s reply email was pretty typical, asking me for more information (which I had to get by opening up the email THEY sent me, so a bit annoying). 

BUT… Then they sent me a survey asking about my experience with Travelocity.  One of the questions on the survey was “Do you know about Travelocity’s guarantee that your booking will be right, or we’ll work with our partners to make it right, right away?” I actually didn’t know, but this question clearly did NOT describe the experience I just had. 

Travelocity did make it right – it took them about 30 days to do so – but that wasn’t what their guarantee said and their survey pointed that out to me.  By including that question in the survey, they planted a seed that they didn’t want to plant and I ended up being less happy overall than I was before the survey.

Tip 2: Ask at least one question that participants actually WANT to answer

It’s important to ask the questions your customers are actually interested in answering.  Too often, marketing departments are so focused on the company’s newest offerings that they ignore the products their customers have come to depend on.

I use QuickBooks for my business. I have LOTS of feedback for Intuit on the core QuickBooks product, but they never ask me about that.  They constantly survey me, asking if I want to buy checks or do payroll or take credit cards, but there is no “Thanks for your time, is there anything else you’d like to tell us?” that would enable me to give them the feedback I WANT to give them.  

Another example from my own professional life: I use Zoomerang for my surveys. It is a GREAT product – with a few caveats.  One of the problems I have with the product is that they have a horrible interface for “choose the answer that most closely applies.”  It’s a small button that’s almost impossible to see. On the other hand, their interface for “choose all that apply” is great – a nice square with a big check mark.  I want to tell them about this issue, but they keep sending me web surveys about other things and I’ve never had a chance to give them important feedback that I really want to give them. Maybe they’ll read my blog and I’ll get to them that way?  <CORRECTION:  They did in fact just send me a survey this week that allowed me to give that feedback.  I’ll wait and see if they act on the feedback.>

Tip 3: Reward your participants, wisely

This tip is inspired by Patricio Robles: “offer users who respond to a survey a discount, an entry in a drawing for a prize, something of value. It will boost response rates and make them feel like they’re investing their time wisely.”

While I generally agree, I would add that when offering a reward to participants, it’s important to consider your target market. A gift that is too nice motivates people who aren’t qualified to complete the survey. Then you have to wade through junk or set up lots of qualifying questions to weed them out. If the reward is nice enough, some people will game the system and try to guess what you’re looking for, so the filtering questions won’t always work. 

When you’re doing a technology web survey, this is especially important, since people who don’t know your topic can really throw off the results, as they are not informed.  If possible, giving a copy of the final report on a topic is a great incentive, since only people who know and care about the topic will respond to this type of reward, keeping your input very clean.

“Technology Expert”, Defined

by

Dimensional Research clearly stands apart from other market research firms because we are technology experts, performing market research for technology companies.

But what exactly does it mean to be a “technology expert” when it comes to research? Obviously it doesn’t mean we’re deep technical experts in every possible area of corporate IT. We don’t write code or manage developers. We don’t manage data centers (although we do our own IT and can commiserate with some of the pain!). We don’t personally install patches or upgrades or test new releases of enterprise applications.

What we’re experts on is Corporate IT and how it works. We are very familiar with Corporate IT and its processes and can talk comfortably and knowledgeably about:

1. Technology. Not the actual coding and implementation, but the types of technology and the tradeoffs.  We understand the perceived benefits and potential downsides of a service-oriented architecture.  We know what cloud computing is (or at least, know all the ways the industry uses the word) and what is real and what is vision.  We understand the challenges with data and the effort that is needed to generate simple graphs and reports.  We know the TLAs (Three letter acronyms) – from IDE to BSM to SDL to SAN (Both the storage one and the security one!)

2. Processes. We know that ISVs and SAAS companies purchase differently than financial service organizations and that those are different than government and non-profits.  We know the software development lifecycle, the people involved, and the products used. We are familiar with ITIL and know how some organizations treat it as a guiding principle, others as one of many guidelines, and some just ignore it.  We know the new focus on costs and services in ITIL v3.  We also know IT has other systems – Six Sigma, COBIT, and more that need to be considered

3. People. We know the common organizational interactions that take place between IT and the business owners; between applications and operations groups; between development and testing; and between server and network teams.  We love talking to engineers and technologists about their challenges, and appreciate the creative side of a job that is often considered to be purely technical.

Most importantly, because we have this foundation, we can quickly come up to speed on any new technology. 

  • On a recent security project we had to learn new vocabulary like “pen testing” and “cross-site scripting”. It didn’t take us long, because we already knew the development and testing processes and quickly understood where security fit into those.
  • On our first data center automation project, we had to learn about different types of automation.  But the products automated processes we were already familiar with, so it was a very straightforward learning curve. 
  • When we did our first cloud computing project, our foundational understanding of virtualization, systems management, and application management meant we were able to be productive right away.

Being technology experts means we “get it.” Whatever technology our clients are developing, whatever processes they use, we are very comfortable talking with IT using professional terminology, asking the right follow-on questions, and gaining the respect of the participants. The result: an efficient and focused market research process that generates a highly effective analysis.

Market Research During Recession

by

Is the market research industry recession-proof? Ray Poynter thinks it’s not. He says, “I have been blogging, reading Tweeting, and engaging in online discussions to try and estimate where research is heading, and I think the story is depressing.”

Mr. Poynter goes on to say that the only reason Market Research has been doing OK so far despite the recession is that companies are still using what’s left of their budget, but this will dry up in April; and that companies are cutting the heaviest expenses first and have not yet reached the stage of cutting smaller expenses such as market research.

I disagree, especially when it comes to Market Research for technology companies. In the sub-market of companies who sell to corporate IT, we’re seeing a very different trend:

  1. Market Research can be a huge cost saver. Instead of spending $250K on a marketing program, our clients see the value of spending $20K on message validation so they can spend $150K on a marketing program that delivers four times the results of the more expensive one – because the messages are hitting exactly the right buttons. I just completed a message test that included evaluating four possible lead benefits.  We found that three of the benefits were incredibly compelling to all audiences, but to 3 out of 4 of our target segments the fourth benefit was actually off-putting and offensive.  Just by deleting that one “bad” bullet from outbound programs their click-through rate on emails went from .5% to 2.0 %.
  2. It is now more important than ever to stay close to customers and build customer loyalty. A Customer Advisory Board clearly shows the participating customers that their voice is being heard – an important factor in instilling loyalty. Done right, it also gathers key information to ensure every functional area of your company is acting in a way that increases customer loyalty – from the product itself to messages from the marketing team, service offerings and ideas for improved support. In a recent customer advisory meeting we ran, the customers came up with a way to administer licenses that was more convenient for the customer and less expensive for the vendor – a win for all groups. Customer Advisory Boards can increase customer loyalty and drive more business to your company, even during the toughest times.
  3. During a recession, ROI is more important than ever. Prospective customers with tight budgets need credible and unbiased information on your products before they buy them. You will need to give your sales team reliable, third party validation of ROI. Turning that ROI into collateral to prove ROI to prospects is highly valuable. And, technology companies have the unique option of getting more for their buck by combining the market research and collateral budget items.

As Mr. Poynter says, “I have been in this industry for 30 years, and I have seen plenty of people lose their jobs and companies go bust. But I have also seen some companies do well, under any circumstances. Some research companies will grow, despite the next two years.” I believe that technology market research companies are in an excellent position to not only survive the recession, but to grow and flourish despite the rough economy. 

Tips For Engaging Customers In An Ongoing Customer Advisory Board

by

Getting your customers to participate in a customer advisory board isn’t too difficult: customers love to sit on advisory boards.  They appreciate not only the opportunity to be heard and to have a relationship with a key vendor, but also the opportunity to hear what’s happening with their peers and to make important industry connections with your other users. 

However, ongoing advisory board programs do require a level of commitment in order to develop a great group dynamic. Typically we ask for a one-year commitment which includes four quarterly meetings.  My personal favorite is to kick the year off with an in-person customer advisory board meeting and then conduct three online meetings.

The best way to keep customers engaged with such an ongoing advisory board program is to show them that their feedback is actually impacting the company and its products. One way to do that is to start all customer advisory board meetings, except for the first one, with a summary of how the feedback from the previous meeting has impacted the company and its products.

But you can go even further and engage your customers even more. When I ran marketing at Freshwater Software, we had a product-focused advisory board (the “ThinkTank”) that was held bi-annually.  Participants were asked to come prepared with their top three product requests. During the meetings, we spent time reviewing those requests and prioritizing them. 

We used to end the meetings by voting on the top three features the group wanted.  We committed to customers that if they took the time to participate, we would guarantee that at least one of their top three features would get into the product. We were often able to get more than just one of their top features into the product. But the important point is that this kind of commitment drove active and passionate participation in the advisory board meetings.

In-Depth Interviews, Focus Groups, or Both?

by

When doing qualitative research, we need to decide which is right for the client: in-depth interviews, focus groups, or maybe a combination of both. This depends on the client’s goals.

When are focus groups better?

Dimensional Research recommends focus groups when the client wants to gain multiple perspectives in an interactive group setting.

One of the main benefits of focus groups is that they get the participants brainstorming. When one participant’s comment feeds off of another comment and so on, the group can really dig deep into an issue. When trying to evaluate market acceptance, capture challenges and issues, or understand objections to new technologies or processes, the focus group dynamic is ideal.

Focus groups have another great benefit – the client can sit behind the glass or on a conference call and hear the direct, unfiltered feedback of a large number of participants with no distractions. Focus group sessions are also recorded for further observation.  If your goal is to expose the maximum number of your team to direct input from the market, this is a very efficient way to do it.

When are interviews better?

In-depth, one-on-one interviews with technology professionals can be conducted in person or over the phone. These are appropriate when the client wants to identify detailed perceptions, opinions, beliefs, and attitudes.

In-depth interviews are particularly effective in the following scenarios:

  1. When the client’s goal is to capture feedback on experiences that occur in an isolated way, such as product messaging or product usability testing.
  2. When there is anything sensitive about the feedback that participants may not feel comfortable sharing in front of other people. They may be concerned a competitor is also be attending the focus group. Or in a win/loss analysis, a customer may not feel comfortable sharing details of poor account manager performance if they suspect the rep might be “behind the glass” and could hear him. The customer would feel much more comfortable sharing this information confidentially talking only with the researcher.

Two things that should NOT drive this decision are:

  • Travel – Sometimes the decision is driven by the geographical locations of participants. If you’re doing customer research with customers located in diverse locations or if you want global representation in the groups without the cost of travel, you still have a choice. The one-on-one nature of a phone interviews is an easy option, or for the group dynamic choose online focus groups.
  • Cost – An important consideration, of course, is cost. As a rule of thumb focus groups and interviews cost about the same per participant, so with the exception of travel – not a consideration for phone interviews or online focus groups – cost should not be the driving consideration in choosing the research approach.

Proving Your Customers’ Successes

by

Clients often tell me, “I have lots of happy customers but none of them will say anything about my products!  How do I get the story about my products out?”  This is a classic pain for anyone who sells into Corporate IT.  You talk to your customers all the time and they tell you how happy they are, but when your corporate marketing team asks for those stories, even the happiest customers won’t let you tell them.  They have their reasons, of course: they may have internal policies, or perhaps they just don’t have the confidence to work with their own corporate communications teams for permission. 

So what you can do? Prospective buyers do their homework before investing in a new technology or product. They seek out third party validation: first-hand experiences from similar companies who have already implemented the same technology in similar real-world environments. Credible, unbiased information allows these decision makers to make confident purchase decisions.

Telling decision makers vague stories about your customer success without backing them up with credible proof just won’t cut it. Your stories simply aren’t that compelling for the IT executive or CFO of a public company that has been burned in the past with products that don’t work. 

A technology market research company like Dimensional Research can bring third party validation to those stories.

We interview your customers, understand their use of the product, get hard and soft data on ROI, and capture some of their best “tips and tricks”. Our technical background means we can understand and capture the real-life benefits your customers are receiving from your products. Then we can turn that into a report that does not name those customers or their companies.  It gives the customer the opportunity to tell the story without jumping through hoops getting permission, and gives you a reliable third party validation that you can use as a sales tool.

Is a Web Survey Right for Me?

by

Dimensional Research’s web surveys are one of our most requested services, for good reason. Many research companies often dismiss online surveys simply because they’re not as profitable. But while web surveys do have several drawbacks, they can be highly useful. As long as the disadvantages are understood, and Web-surveys do not become the only research tool in the toolbox, Web surveys can yield very powerful results.

So what’s WRONG with Web Surveys?

Web surveys are a quantitative research vehicle. As such, they are highly efficient, but easily miss contextual detail. They make it impossible to capture subtleties. When using online surveys, the researcher doesn’t get an opportunity to drill down into the research topics, to ask the participants clarifying questions or to uncover hidden issues.  The all important research tools of “why” and “tell me more about that” aren’t available.

A big challenge in crafting web surveys is question design. Designing the right questions is incredibly important because once the survey is fielded, the researcher doesn’t get the opportunity to discuss the questions further with the participants, to clarify how words are understood or to capture the subtleties of the answers.  Unlike qualitative research, the survey is only as good as the question-writers understanding of the problem.  There is very little opportunity to learn anything beyond the scope of your current knowledge.

Then, what’s GOOD?

Web surveys are an inexpensive, cost-effective way to get a broad range of input. They can be executed quickly to reach a large number of respondents in multiple geographies. Internet surveys enable researchers to collect a large amount of data in a relatively short amount of time, and to draw responses from all over the world, resulting in a dramatically larger sample size than they could get using qualitative research methods.

With today’s online survey tools – Dimensional Research uses Zoomerang and Survey Monkey –  reporting is immediate.

It is relatively easy to recruit participants for web surveys because they involve minimal time commitment. This is especially true for participants who are technology professionals, since these participants are very comfortable with online tools.

So, back to the original question – is a Web Survey right for me?

Maybe.  If your goal is any of the following, Web Surveys should definitely be considered:

  1. Capturing the hard data you need in order to capture customer pain points, support your business plan, or prioritize product features
  2. Documenting market conditions for use in outbound marketing efforts.
  3. Gathering information quickly when time is the most pressing consideration.

To Recruit the Right Participants Start by NOT Recruiting the Wrong Ones

by

Recruiting the right participants is one of the most important aspects of a successful market research project. We strive to fill our projects with participants that struggle with the issues our clients are working to solve, and that are eager to participate so they give thoughtful feedback. Above all, we avoid participants that are not the right fit for the project. The key is to develop a recruiting guide that ensures that those who meet the specifications are the right people for your project.

Sounds kind of obvious, right? Unfortunately, this can be very difficult. Clients can always tell us who they WANT us to recruit. It is our job to drill down into those specifications and understand who they DON’T want us to recruit.

For example, let’s assume the client is targeting software developers. It turns out that different types of developers have very different purchasing behaviors. If we just say “software developer” as a criteria for participation, we may get the single entrepreneur in a garage developing tools sold via word of mouth, alongside a senior architect from a financial services company responsible for a mission critical online trading application, next to a low-level firmware developer for printer software – probably not similar pain points!  It’s important to pinpoint, prior to starting the recruiting process, which type of developers we want to recruit, and filter out everyone else.

If the goal of the project is to decide which kind of developers your product is most suited for, then of course you want a wide range of participant types and will intentionally recruit a mix with broad representation. In this case, it is important to recruit several of each kind of participant so there is enough feedback to correlate responses at the end.

Obviously, expertise in technology is highly important for a targeted technology recruiting process. We have worked in the IT corporate world and are highly experienced and understand technology – so we  know how to write a productive recruiting guide that weeds out the wrong participants.

Technology Market Research Is Best Done By Technology Experts

by

Anyone can do technology market research, right?

Wrong. When dealing with IT professionals, it is very important to actually understand technology, for several reasons:

When the researcher understands technology, she gains the respect of the participants. As a result, they cooperate with her and share more of their own knowledge and expertise. The resulting marketing analysis is highly focused and useful.

When the researcher is a technology expert, the participants are comfortable using professional terminology and speaking at a deeper level of technical detail, because they know the researcher will “get it.” Again, the result is a highly effective marketing analysis.

A moderator who has expertise in technology can ask the right follow-on questions, because he understands what the IT professionals are talking about: Let’s imagine a scenario where a moderator is talking to a security professional about network security. The security professional unexpectedly starts talking about application development. Many moderators would not understand the connection between developers and security experts, and would have missed the opportunity to learn more about this important relationship.

A moderator who is a technology expert is knowledgeable about the application development lifecycle – past, current, and future. Such a moderator will drill down into exactly how when exactly the security professional works with the developers, and use the knowledge he gains to create actionable recommendations.  This is not a theoretical scenario: the key to success in a recent project I worked on was understanding the involvement of both security and development teams in the purchasing process.  Understanding the workplace dynamics between the two teams allowed me to recommend a go-to-market strategy that focused on the diverse roles these different players had.

Another great example from my own professional experience of using technology expertise to produce better, more accurate technology marketing analysis: a moderator who was comfortable with ITIL enabled a group of participants from Europe to use the ITIL language to quickly communicate their issues. No time was wasted with this group of busy IT executives explaining basic concepts such as the difference between incident management and problem management. Since the moderator was familiar with these concepts, the conversation was highly efficient and focused on pinpointing issues and brainstorming solutions rather than on explaining fundamentals.