Getting Better Data From Ranking Survey Questions

In my last article about survey question wording, I mentioned I would talk about the different methods in a survey for getting a ranking of a list of items. Such a ranking has broad applications in several areas. In marketing, it helps identify the most important product features. For customer satisfaction, it guides decisions on which improvements to prioritize. In employee satisfaction, it determines the most critical aspects of job satisfaction.

So here we’ll look at some detailed examples of how you could get a ranked list without using a ranking question.

Ranking Survey Questions Defined

Before we dive into some specific case studies, let’s first explore briefly what “ranking” means in the context of a survey.

If you are my only customer and I ask you to rank five possible criteria by importance, your response completes the task. Your opinion is the only one that matters, and I have all the information I need to evaluate each criterion.

However, suppose I have two customers. Is it still best to ask each to rank order the five criteria? What if I have three customers or more? How do I use this data to “rank” my criteria?

When I have one customer or one critical party, ranking gives an unambiguous ordering. That changes when you are looking for a ranking from a group of survey respondents. The result of rank ordering doesn’t translate directly from one to many.

In the above example, we could argue that criterion D deserves the number one spot and criterion C should be ranked second. However, beyond these two, the ranking becomes unclear.

Ranking Survey Question Example: Herbal Tea

Let’s say we want to rank the criteria herbal tea buyers use when buying tea. We want to know the best ordering of the following buying criteria for herbal tea drinkers:

• Clinically tested
• Environmental practices
• Price
• Flavor
• Certified organic
• Effectiveness
• Fair trade certified

Knowing our customers’ priorities will help guide engineering and marketing efforts.

We have already talked about one way to approach this problem: we can ask each respondent to rank order the seven criteria. This is similar to a ranked choice survey, where respondents can provide their preferences in a structured way.

Now let’s look at two other alternatives.

Survey Question Ranking Alternative #1: Top Choices

The first alternative is to ask respondents to select their top two or three criteria out of the list. (A good rule of thumb is to have them limit their selection to about one-third of the total number of items.)

This requires the respondent to do a similar evaluation as he or she would in rank ordering the criteria, but the evaluation is not as extensive. The results yield a natural ranking by looking at the criteria according to how many “votes” each item received.

Survey Question Ranking Alternative #2: Distributing Points Among Choices

The second alternative is to give respondents 100 points to distribute among the seven choices as they please. This option also requires a similar evaluation from the respondent. However, it offers more flexibility by allowing them to divide the points among all the items. The only requirement is that the sum of points assigned adds to 100.

To summarize, here are the three choices to get a ranked list from a survey:

1. Rank order the criteria from 1 to 7, based on the importance they have on your buying decision (ranking question example).
2. Select the three most important criteria for your buying decisions (ranking questions in surveys).
3. Distribute 100 points among the seven criteria based on the importance each has on your buying decision.

Now let’s compare the three options so you can decide which one works best for your particular survey needs (we’re using a list of 7 criteria for these examples, but you could have any number of options).articular survey needs (we’re using a list of 7 criteria for these examples, but you could have any number of options).

Survey Question Options That Benefit Survey Respondents

#2 is clearly the easiest to answer for the respondent. The respondent merely has to divide the seven criteria into two groups: important or not important.

#3 is the next most friendly to the respondent because the respondent has the flexibility to decide the level of effort they want to put into the selections. If a respondent is so inclined, he or she can select his or her top three choices like alternative #2 and then just divide the 100 points among them. However, with this alternative, the respondent has the flexibility to assign different point totals to the one or ones he thinks are important.

The respondent must put in the most effort for #1 because they have to assign a separate rank to each criterion. This becomes challenging when two items seem equally important or when the bottom of the list includes three or four equally unimportant items that still need to be ranked.

Ease of Analysis and Interpretation for Options

Suppose we have 100 responses to the ranking question. For alternative #1, this yields 100 sets of 7 numbers (1 to 7). For #2, we have up to 300 “votes,” and for #3, we have 100 divisions of 100 points into seven groups.

#2 and #3 are easily summarized and interpreted. One would just total the “votes” or “points” for each criterion to create a natural ranking of the seven criteria. For example, survey ranking questions like these can quickly show which criteria are most valued by respondents.

#1 is different. The summary is not so straightforward. I could average the rankings for each criterion, but I have to decide if I can live with a scheme where 100 4’s is the same as 50 1’s and 50 7’s.

Alternatively, I could total the number of times each criterion got a first-place rank, but then I have to be happy with a scheme that gives a higher ranking to a criterion with 10 first-place ranks and no others in the top 3 than to one that got 9 first-place ranks and 91 second-place ranks.

Analysis and interpretation for option #1 takes some thought and rarely has an easy answer.

Which Ranking Option Should You Choose for Your Survey Question?

I like the simplicity and ease of analysis and interpretation of the select-your-top-one-third method (#2). It provides a natural ranking and is unlikely to overwhelm or frustrate the respondent. It is important to keep the respondent in a good frame of mind for answering the rest of the survey.

Clearly, the distribute-100-points method (#3) provides more precision. If I feel like I need more precision in distinguishing between items on a list, I might use this method, but in most cases, it is not really needed. Remember, it is always a good idea to consider how you are going to use the information you get back when deciding on a question to ask. In this case, one might want to consider what kind of precision is warranted.

Maybe we should rank order the alternatives!

In summary, understanding ranking questions in surveys and applying effective survey ranking techniques can significantly improve the quality of your data, particularly in market research contexts where insights into preferences are crucial. Examples of ranking questions can enhance your survey design and lead to better decision-making outcomes.