When designing survey questions, selecting answer types and creating answer choices, there are a number of things that need to be considered in order to attain high quality data. The way you ask your questions and how you let participants respond plays a huge factor in how thoroughly you will be able to analyse and compare your data. This will impact the resulting accuracy and validity of your conclusions.
Answer Scales and Subjectivity
Scales help people answer questions quickly with a value that you can then analyse easily as quantitative data. Asking someone to rate their experience or asking them how often they do something, without giving a list of options to answer from, opens up the way they can answer very widely. Therefore, people can get easily confused and may either throw out an answer without thinking properly or might derail onto explaining how long they had to wait for the bus a couple of days ago. Scaled, predefined choice help, but you need to make sure that your choices will lead to high quality results, otherwise you would have been better off letting them answer in their own way.
For measurable questions: Keep your answer choices objective and easy to quantify
Let’s say you are trying to do some market research for the development of your new online shop and you want to explore how frequently your buyers make use of other online stores with offerings from your same industry. In order to answer the question you could provide them with a scale such as; very often, often, sometimes, rarely, very rarely. However, what will happen here is that your results will be of very low quality due to the subjectivity of what ‘often’ really is. Take someone who’s friends use such online shops every day. If they themselves only use them once a week they might respond with ‘rarely’. On the other hand, someone who uses these competitor’s online shops once every two weeks but surrounds themselves in circles of people who don’t shop online at all, might respond with ‘often’. What has happened here is that due to the subjectivity of the scale, the individual who uses the online shops less than the other actually rated higher on the scale, rendering your data practically invalid.
What you can do is to, whenever possible, provide a scale that is measured against something, such as time. An effective sale in this situation would have been; almost every day, at least twice a week, up to two times per week, about once every two weeks, less than once every two weeks.
For subjective questions: Keep your answer choices balanced
When your question is not about something objective like frequency, weight or age, you will need to take the balance of your answer choices into consideration. When an answer is subjective, about the perceived value of one’s experience, for example, you should never have more positive than negative answer choices and vice versa. This is to avoid having positively or negatively skewed results. The inclusion of a neutral answer is very much dependent on what you want to analyse and whether you need to force respondents into deciding positively or negatively. The availability of a neutral answer makes it easier for respondents to avoid making a decision. Without a neutral response, your scale would look something like this: Strongly Agree, Agree, Slightly Agree, Slightly Disagree, Disagree, Strongly Disagree. If you are using numbers as answers, include an even number of answer choices (ex: 1-10) to avoid neutral answers, or an odd number (ex: 1-5) to include a neutral answer.
Keep your goal in mind
Subjective questions such as, “How was your overall experience on a scale of 1-10?”, are not about what actually happened, but about how the respondent felt during the experience. Two respondents may have received the exact same service in exactly the same way. Due to different personal backgrounds and past experiences however, one may rate their perceived experience as a lot better or a lot worse than the other. Therefore, if you are seeking to analyse actual behaviours and occurrences, you will be better off asking a lot more objective, quantifiable questions.
Avoiding Skewed Results and Obtaining High Quality Data
In Part 1 of this article, we discussed the delivery method of your survey. Unless you are conducting a survey with a very narrow audience for whom one particular method works best, there are so many variables that come into play when considering how to minimise the chances of skewed results.
Survey delivery methods
For one, if you conduct your survey purely by sending it out via email, the results you gather will likely be gained from individuals who are less apathetic towards reading, meaning that the responses gathered will likely be from people who have a higher level of education or social status than if you conducted a telephone survey. On the other hand, conduct your survey only by phone and you might miss out on the opinions of people who are constantly in and out of meetings, but will gather a lot more responses from people who spend more time at home or those who have time to kill. Fort this reason, you may want to consider conducting your survey through different mediums simultaneously.
Surveyor and question bias should also be avoided as people are more likely to tell you what you want to hear. Therefore, it is highly important for the person asking the questions to remain objective and to not consciously or subconsciously prompt the individual to answer a question in a certain way. This is equally important for questions with pre-defined choices and for open-ended questions such as, “How would you describe your exciting experience today?”. This question wording will skew your results as your data will likely contain a higher number of responses that included the word ‘exciting’.
Doubling up your questions
You should always avoid asking two questions in one. For example “How would you rate the friendliness and helpfulness of our staff today?”, should be split into one question about the friendliness and a separate question about the helpfulness of the staff. This way, the quality of your data will be higher and will create more accurate results about their attitude at work.
If everyone started conducting clear, straight to the point, ethical surveys, then maybe it would be easier for organisations to gather honest data when they really need it, enabling more effective betterment and development of customer service, employee wellbeing, organisational offerings, and much more. This makes it all the more important that you are geared up for conducting surveys in a way that allows participants to complete them without regretting taking part. In the end, there is a reason that the survey was conducted. The surveyor cannot read minds and needs the participants to state their honest experience, without which development becomes a bit like shooting in the dark. In the end, who doesn’t want to go to work happier, leave a shop smiling, or find new products on the market that really hit the nail on the head when it comes to fulfilling our needs? So be kind to your participants and make the most out of your survey time together.