If you’re reading this, you’re likely hunting for ways to step up your customer survey game. You’re in the right place, but we’re not talking about success stories today.
The opposite, actually.
What follows are common survey mistakes that can skew your data for the worse, and solutions to minimize the damage they can do.
Dollars for your Thoughts?
You’ve probably encountered an online platform that provides rewards to users in exchange for completing surveys. Some pay out cash, but most use brand-specific reward scales like loyalty points or specialty items.
Sounds good at first, but there are a few potential problems.
For one, surveyors often change what they’re looking for to get a broader sample for their results, which is fair practice. However, users will often create dummy accounts to avoid being disqualified, gaming the system and grinding out rewards nonstop.
For another, users can take surveys, finish, and not receive their reward to compensate them for voluntarily providing valuable data. Not only that, surveys can range from 5 to 30 minutes long. In our humble opinion, a survey exceeding 5 minutes is almost criminal.
This is what a typical survey reward interface looks like.
I’d like the point out that incredibly small text at the bottom.
Big red flag. The developer can now ignore complaints regarding the fact that they knowingly host third-party surveys that can either waste your time and take your information or worse, contain malware.
It’s the classic “use at your own risk” disclaimer. But there are two problems here:
1. The text is ridiculously small. As in, some people who don’t need glasses might have trouble reading this.
2. There’s no voluntary contract agreement at any point in the navigation, no button saying “I agree” or anything of the sort, which under the GDPR, there definitely should be, particularly where user consent is concerned.
This is shadier than wearing sunglasses under an oak tree on a cloudy day in Great Britain. It may even be illegal, depending on where you do business.
These seemingly shallow potholes can turn into pitfalls that hurt the reputations of both the survey platform and their business partners.
Solution: Thoroughly vet your survey platforms. Better yet, conduct surveys internally to avoid the hassle. It’s worth the effort.
The Danger of Misusing Metrics
You’ve almost certainly dealt with NPS®, CSAT and other buzzword-y sounding metrics. They’re everywhere, and for a reason: they’re pretty useful. And the color-coded graphics make for compelling boardroom presentations.
But there’s a danger lurking in their misuse.
Have you ever had someone ask you to fudge their rating even though the experience was sub-par, or because their livelihood depended on it, or both?
You may think that’s just low-level employees being pushy and self-serving. But the actual culprit is misplaced emphasis on the metrics themselves, whether NPS® or others.
It typically starts at the executive level, when bonuses are tied directly to raising a specific metric. Those execs then pass that pressure on down the employee ladder until it trickles down to survey administrators at the service level.
Who better to point out this problem better than the creator of NPS® himself?
“First, linking NPS® to compensation focuses the organization on the score as the objective in and of itself rather than as a tool to ensure that people learn the right lessons and take the right actions to improve CX. Second, it puts inordinate pressure on the team developing the measurement process to get the metrics right. And third, the direct link to bonuses – whether in the C-suite, sales or call centers – will almost certainly encourage gaming and manipulation.”
– Fred Reichheld
Solution: Remove employees’ ability to influence the results entirely by employing an impartial client advocate, as suggested by Satrix Solutions.
Heuristic Holes and how to Close them
If you know some economic psychology, you may be familiar with the work of Daniel Kahneman. His studies on heuristics (think mental shortcuts) and cognitive biases are pivotal to survey-based research and statistical analysis, and an invaluable resource in the science of decision-making.
Heuristics are a tough hurdle to overcome. Doubly so because you as a company are just as susceptible to them as your customers. We’re all human. We don’t even realize we’re using heuristics most of the time.
She finds that having processes in place to protect you – and your customers – from the short-sighted nature of heuristics is crucial to managing them.
Avoiding cognitive blind spots in customers is key to maximizing the quality and validity of your results. There are a LOT of them. We’ve chosen just a handful to highlight.
1. The Anchoring Effect
Anchoring is when a subject puts too much value on the first information offered when making decisions. A salient example is the asking price of real estate. That initial price – whether it’s 500k or 5 million – will dominate the range in which negotiations are made. Similarly, your respondents can be influenced by unnecessary values or details that appear in your survey.
Solution: Use the KISS method; Keep It Simple, Silly. Being straightforward in your questions and use simple language. Avoid unnecessary details whenever possible.
2. The Halo Effect
The halo effect occurs when a person’s first impression of someone or something is overly positive. This impression will influence other perceptions of that thing as if it applies to all their qualities.
“This person has a strong, confident appearance. I’m sure they’d be a good fit for president” is a painfully common example.
You may run into this when going up against brand identification, and how you or a competitor are already positioned in your customer’s mind, likely on multiple levels. Branded images, sounds, and tone can all interfere with results. You almost have to de-brand your survey to avoid this problem.
Solution: Use a third-party platform that handles actually generating the survey questions and tracking results.
This can greatly reduce both your own preexisting biases and those of your customers at once.
(Just remember to avoid the shady platforms.)
3. Cognitive Strain
This is when the brain is forced to make multiple calculations, and can’t rely on quick decisions. At first glance, it looks like something to be avoided.
But under the right conditions, cognitive strain can actually make the customer more engaged during your survey, resulting in more thoughtful – and more useful – answers. Studies have suggested that introducing strain by making the questions slightly harder to read (blurry font, low-contrast background for example) actually forces their slow-thinking and processing systems into action.
But be warned: don’t overdo it. Too much strain, and the surveyor will take the path of least resistance, that little X in the corner.
Solution: Use cognitive strain in a balanced way, and test it thoroughly before implementation. There’s a definite average threshold where strain is simply too much, and you should find it first.
4. Confirmation Bias
Even if you’ve never heard of the above biases, you probably recognize this one. This is the tendency to interpret new evidence as confirmation of your existing predispositions without considering other information, and is perhaps the most common and insidious example here.
It’s hard to overcome a set belief, and confirmation bias feeds on it.
Solution: Look actively for both positive and negative results. They are equally informative and necessary to your success.
Sounds like a lot to worry about
And it is. But the more we understand these problems, the easier they become to fix. Knowing is half the battle.
Which means you’re already halfway there.
Survey technology is getting more sophisticated by the day, and you should endeavor to give your customers the best possible experience so they can give you their best possible data. There’s a solution for every problem out there; you just have to find it.
Net Promoter Score and NPS® are registered trademarks of Bain & Company, Inc., Fred Reichheld and Satmetrix Systems, Inc.
Junior Marketing Analyst with a passion for Market and Workplace Psychology, Gaming, and Strategic Analysis.