Response option configuration of online administered Likert scales

Surveys and questionnaires are a good way to gather information from a large number of people. More often than not this is done using Likert (side note: FYI, it’s pronounced “Lick-ert”) scales, which consist of a series of phrases or statements that are accompanied by a labelled scale, e.g. from “strongly disagree” to “strongly agree”.

Likert scales are kind of like programming languages: both have been around for quite some time now, but there is still a lot of disagreement about how they should be used and why.

Common issues with Likert scales include:

• acquiescence, i.e. when individuals slightly, but consistently agree or disagree with statements;

• left-side response option selection bias, which makes it more likely that respondents select response options that are presented on the left side;

• negatively worded phrases may negate some of that selection bias, but could also confuse respondents.

Researchers have tried to think of solutions to mitigate such issues, with mixed results. This study takes a crack at resolving the second issue, by comparing the effect of arranging response options in various different ways.

The arrangement of response options can be varied by changing their orientation (horizontal or vertical) or their directionality (“strongly agree” to “strongly disagree”, “strongly disagree” to “strongly agree”, or a mix of the two).

The author conducted a study with 1693 participants from two public universities in the United States. Participants were given one of six versions of the same survey. Each version used a Likert scale with a different orientation and directionality of response options. Information about the purpose of the study was withheld from participants.

The survey responses were analysed using a lot of statistical tools and methods, which I am going to ignore in this summary. Instead, I’ll jump right to the conclusions:

• Existing claims about left-side response option bias were based on Likert scales in paper-and-pencil surveys. This bias is also present in responses from online surveys. Fortunately, selection bias is only really an issue for absolute judgments. If you only need to compare between groups or relationships of multiple variables the bias is the same for all responses.

• Vertically oriented response options seem to eliminate the selection bias. They come with a downside as well though: they take longer and (curiously enough) require more mouse clicks to complete (side note: Vertical options take up more space!). This may have a negative effect on survey return rates.

• Bidirectional Likert scales also take more time to complete than unidirectional scales, i.e. those where all items are accompanied by ascending or descending scales. However, they do not eliminate selection biases.

• Orientation and directionality of response options do not affect the internal consistency reliability of the survey.