An Unanswerable Question

Yesterday I stumbled across another poorly designed question when answering one of the surveys I received.

Poorly Written Survey QuestionThe question reads, ‘Do you live in any one of the following metropolitan areas?’ Possible answers are:

  • London or surrounding suburbs
  • Birmingham or surrounding suburbs
  • Glasgow or surrounding suburbs
  • Liverpool or surrounding suburbs
  • Leeds or surrounding suburbs
  • Other (please specify)

I don’t live in any metropolitan area; I live in a small town in rural Lancashire. My nearest metropolitan area (i.e large city) is Manchester, but at 30 minutes’ drive away, I hardly qualify as living in the suburbs. Therefore, no answer applies to me for this particular question, so I just pressed ‘Next.’ So here’s the rub: I had to answer the question, despite none of the answerable options being applicable to me, as the friendly message reminds me:

Please review your responses on this page. One or more questions require further input. Please select an answer.

A couple of thoughts occurred. First, the survey designers only wanted respondents from these areas to reply (not sure why, but it’s possible). Second, they made a mistake in the logic and setup of the question (which is much more likely in my opinion). Either way, it’s a poorly designed question, and I left the survey feeling disgruntled (and I have more tenacity than most, I would guess).

If the survey did require only people from metropolitan areas to respond, it should have clearly stated this in the invitation. And, if someone missed that and began to answer the survey, this should have been the first question of the survey to screen respondents, with the option to select ‘I do not live in a metropolitan area or suburb.’

If this was a mistake in the logic or question design, then first it should have undergone more testing. I can’t believe such a basic error as this one was missed. Specifically, this question could have been fixed either by adding an option to choose ‘I do not live in a metropolitan area,’ or why not just ask for the first part of the respondent’s post code? It wouldn’t be difficult to work out from there which metropolitan area, if any, they live in.

It is possible to keep the question as it is, and allow respondents who don’t live in a metropolitan area to click ‘next’ without selecting an option, but I think that would be unwise. It would be too easy to get a false negative (i.e. people who actually live in a metropolitan area might miss the question).

What was nice to see on this survey, which many neglect, is the progress bar at the top. This is very reassuring to see, and serves as a reminder of how many questions I have to answer.

Advertisements