InformationWeek did a very interesting article on Practical Analysis: User Habits And Making Tablets Seem More Like Beer. They did a survey showing IT prefers HP, then Dell for tablet computing and only then the infamous Apple iPad. But how can it be possible? “Everyone knows” the iPad is the market leader! The answer, according to the article, is that IT acts very differently than consumers do, and this was a survey of IT professionals.
We didn’t conduct the survey so we can’t comment on the methodology used (although as much of the methodology as the article reveals certainly sounds sound), but we do completely agree with this: When research delivers a finding different to core beliefs, you’d better be able to explain why.
There is almost always an explanation, although perhaps it’s not as obvious as the Infoweek example. We recently did a series of projects that included questions on how various IT professionals use vendor web sites when making purchase decisions. The participants in our study were adamant that they didn’t like videos – they weren’t scannable, tended to be full of “marketing fluff,” and were generally a waste of time. But our clients’ web analytics showed that videos were viewed a lot.
The instinct was for the web analytics people to assume that the research was wrong, because they had real-life data showing that videos were popular. Of course, we had to consider the possibility, since one of the well–documented challenges with research is that people don’t always act the way that they say they will act. But we were pretty confident in our methodologies, and our techniques for getting to actual behavior, so we decided to figure out why. And as they did in the Infoweek article, the first thing we did was look at our demographic. Was there something about the demographic we spoke to that behaved differently than the overall Web audience?
Once we started digging, the answer was obvious: Our study was with product-level decision-makers – the most technical of the technical, the “sharpest pencils in the box,” the brightest ones. They didn’t like videos because they understood things quickly and preferred to scan content. But we discovered these decision-makers also had to educate the rest of their teams – and they would frequently forward videos. Since there were multiple “less technical team members” that watched videos for every “technical decision maker” that didn’t, the behavior was lost in the analytics.
If this kind of “what’s the underlying explanation” problem fascinates you the way it does us, we recommend a great podcast from HBR: Strange-But-True Research Insights.