Dimensional Research

Technology Market Research

Yet Another Useless Customer Survey

by

I’m on the phone right now with my bank, doing a “post transaction” survey.  I always take these – what researcher doesn’t? And once again I’m annoyed.  What has my bank learned from doing this survey?  As far as I can tell, ABSOLUTELY NOTHING.

The purpose of my call to the bank was to get that number you need so someone can wire you money.  Let’s see how the “post transaction” survey went:

  • A question about the banker I talked to.  He was just lovely.  I asked my question badly, because I forgot the name for that number (routing? IBAN? something like that), but got an answer right away.  Gave him a high number.  Good so far.
  • Wait, aren’t you going to ask me about the experience of getting to my lovely banker?  Do you care that it took three calls and five minutes to get to the 30 second conversation?  Apparently not.  No questions about that.
  • Now a question about the range of services offered by my banker.  No idea.  I only asked him one question.  He answered it well.  I can’t answer this question, but “don’t know”  is not an option on this survey. I can only choose “1=unacceptable through 5=excellent.”  Where’s the N/A?  Do I say “excellent” since I liked him and don’t want to get him into trouble?  What if his bonus is based on this survey?  But that will be misleading in the analysis.  I could give a 3 to at least not skew the averages much, or just hang up.  Or do I give him a 5 because I liked that he didn’t try to sell me anything else after he answered my question?   I’ll just give a 4 to split the difference.
  • What?  We’re done? Where is my “If you’d like to leave any other comments please visit www.bank.com/survey” so I can tell them about the last question.  How difficult is it to have a form for follow up in this day and age?

I will stop now before I begin ranting, but once again I feel like today’s ease of implementing surveys on web sites and telephone is causing some of the worst market research that has ever happened, and in the process annoying people who might stop doing surveys that are actually well designed and will drive better business practices.

Customer Satisfaction Surveys: Avoid “Survivor Bias”

by

One of the biggest mistakes in doing customer satisfaction surveys is allowing “survivor bias”.  This happens when you surveying your existing “surviving” customers – the ones that stayed with you no matter what – and ignoring the customers and business that you lost.

“Survivor bias” refers to drawing conclusions only from data that is available or convenient and thus systematically biasing your results. In technology market research, it means neglecting former customers and only drawing conclusions from existing customers – which are by definition happy customers (or at least happy enough that they have not stopped doing business with you).

If you really want to understand customer satisfaction, don’t just get feedback from the people who continue to be your customers no matter what you do to them. You need to include your former customers – those who have turned to your competitor or just plain don’t use the solution anymore.

Allowing survivor bias is a particularly heinous research mistake because it’s not hard to avoid.   The data is there after all – if you have data on your customers, you have data on your former customers.

This raises one very important question – do you even know that they are “former?”  The research can help you find that out, and even more importantly, find out why they are no longer customers. If there’s a problem with your solution, you need to know about it, and the sooner the better.

In this year’s annual customer satisfaction survey, make an effort to include former customers. After all, as we said here before, your goal is to get honest feedback, and when it comes to market research, negative feedback is good for you!