Anyone Can Write a Survey, but they Probably Shouldn’t…
In almost every industry there’s a skill that everyone thinks is easy. It usually isn’t. This doesn’t stop them from saying “I could do that” and then proceeding to do it poorly. In research, that skill is almost always writing surveys.
Somewhere in the distant past, I used to teach research methods to college undergrads – this included a brief introduction to survey writing that probably took a week out of the semester. The goal wasn’t to equip them to actually write functional surveys, but rather introduce them to some of the technical components of surveys and get them thinking about how to approach research questions. The people telling you how easy it is to write a survey? They usually don’t have even that level of training. And if someone with no training is writing your surveys, your research might be bad.
Often, the first things to go are the technical aspects of survey writing. Sure, you get poorly operationalized and overlapping response options, but these are relatively easy fixes. Another signature of the overly enthusiastic armchair survey writer is the leading question. The reason for this often makes sense: organizations who believe in their product or mission have a bias toward the positive. This translates into survey questions like “How outstanding was our service?” or “Given the exceptional quality of our product, would you purchase it again?”
Another common technical flaw, frequently in response to a feeling that the survey is getting too long, is the use of double-barreled questions. These are items that ask questions like “Were you satisfied with the timing and location of our conference?” or “How frequently do you purchase our products and how much do you usually spend?” These technical flaws aren’t insurmountable, but they do make bad surveys.
Beyond technical issues, another overlooked aspect of survey writing is the conceptual elements underlying both the survey items and the research project itself. Conceptual flaws manifest in a variety of ways:
Semantics. Are your questions asking what you want them to? Will respondents interpret them the way you do? Just because you know what you meant to ask doesn’t mean anyone else does.
Speculative questions. Are you asking about something someone hasn’t done yet or what they think they might do in the future? People are terrible at predicting future behavior.
The kitchen-sink approach. Do these survey questions serve the purpose of your project, or are you just asking everything you can think of?
Inside baseball. Are you using industry jargon or trying to deploy some sort of conceptual rebranding? Don’t do this in your surveys. Your organization might call it an “optimized feline nutrient supplement” but if consumers think it’s “cat food”, then that is what you should call it.
Why is this bad? It’s a straight-forward equation:
Bad surveys —> Bad responses —> Bad insights
Creating surveys isn’t just about writing questions and creating items that will deliver accurate data and actionable insights. It also takes time and practice. If your project is worth doing, then it is also worth investing the appropriate resources to get it done right. Otherwise, it's just producing bad research.