| Most
problems with questionnaire analysis can be traced back to
the design phase of the project. Well-defined goals are the
best way to assure a good questionnaire design. When the goals
of a study can be expressed in a few clear and concise sentences,
the design of the questionnaire becomes considerably easier.
The questionnaire is developed to directly address the goals
of the study.
Step 1: Determine how many questions to pose.
Include enough items to indicate a seriousness
of purpose but not so many items that they overburden the
respondent. Although research indicates that shorter questionnaires
usually generate higher response rates, questionnaire length
by itself probably isn't as important as other factors we
will discuss. There is little benefit to shortening an already
short questionnaire (say, reducing 8 items to 4 items), because
this may signal that the task isn't very important.
Reducing an 80-item questionnaire to a 40-item
questionnaire may improve the response rate, but the difference
is not likely to be huge (perhaps a gain of 10-15% more respondents).
There is some evidence that respondents balk at questionnaires
that exceed 100 items.
Step 2: Order questions to maximize interest
and maintain logical flow.
From the viewpoint of the respondent, the
response to each item is influenced by the items before and
after it. There are two general considerations for shaping
the order of questionnaire items.
A) Begin the questionnaire with items that
are easy, appropriate for all respondents, and central to
the main topic; put sensitive items at the end. The respondent's
reaction to the first few items usually determines whether
he or she will try to complete the questionnaire. It is important
to begin with one or more items that all respondents can answer.
Thus, if you are asking teachers about their instructional
resources, you wouldn't start with questions about computers
because only some teachers may use computers. In addition,
the questions should be easy in the sense that respondents
should not have to obtain a lot of information to generate
answers.
The initial questions should pique the respondents'
interest and engage them with the questionnaire. Begin with
items asking for opinions on matters likely to be of interest
to them. For example, if you are interested in teachers' instructional
resources, you might ask first how satisfied they are with
the instructional resources available at their school, since
they are likely to have strong feelings about the matter and
will welcome the opportunity to express their opinions. For
the same reason, do not start the questionnaire with items
about background characteristics (e.g., number of years teaching),
because such questions, although easy, will not engage them
much. Save them for later in the questionnaire.
After items most central to the topic are
presented, subgroupings of items more narrowly focused or
less central to the topic should follow. If the questionnaire
includes questions to which respondents may be especially
sensitive, such as salary information, they should appear
near the end because respondents are less likely to quit when
nearly finished.
B) Group items into logical subtopics while
maintaining interest. Deciding how to order all of the items
for a questionnaire requires balancing several concerns. On
the one hand, respondents typically give more meaningful answers
if questions flow in a logical order, with items on a given
subtopic grouped together. On the other hand, if there are
too many items in a row about one subtopic, respondents may
lose focus and motivation.
Therefore, one needs to find a middle ground
between asking questions in a completely random order (likely
to frustrate respondents) and asking too many narrowly focused
questions in sequence (likely to make them lose interest).
If one ends up with too many items in a subtopic, one solution
is to try to reduce redundancies by crafting more inclusive
wordings or eliminating less productive items. Also, if items
in a subgroup share the same response categories, an items-in-a-series
format is efficient (see Figure 1).
Step 3: For individual items, select a consistent
item style and emphasize navigational cues.
Once items are ordered sensibly, consider
each item's graphic style and the navigational cues available
for working through the questionnaire. Style includes factors
such as typeface and typesize, spacing, and appearance. These
factors strongly affect respondents' ability to understand
what is being asked and to respond appropriately.
A) Item style. There is no single "best"
item style, but the following guidelines address common issues
with predetermined-choice items.
Run answer choices vertically rather than
horizontally; a vertical arrangement is both easier to fill
out and to score.
The exception to this rule is items-in-a-series
questions, for which the answers have to run horizontally
because it is the most efficient way of grouping the questions
together.
Differentiate the question from the answers
by using different font sizes and weights. A good technique
is to use a bold font for the question, followed by an unbolded
font for the answers. However, keep font variations to a minimum;
it is easier for the reader if the entire questionnaire is
done in a simple, consistent font style.
Use a consistent approach for answer options.
This involves presenting items that involve a range of options
in a consistent order. For example, if "strongly agree"
is given as the first option, then the other agreement-type
items should start in the same way. Respondents also should
be able to mark all their answers in the same way. Thus, don't
mix checkboxes with circles or underlines.
If the respondent is being asked to write
out an answer of at leaset several words, draw lines and leave
adequate space between lines to encourage legible handwriting.
B) Navigational cues. Long questionnaires covering a variety
of subtopics can be challenging to respondents, particularly
if there are multiple sets of directions and places where
a respondent might be asked to skip a series of questions,
depending on an answer. The following techniques minimize
the burden put on the respondent.
Number all items consecutively, avoiding letters
or number-letter combinations. Even if the questionnaire has
distinct sections, continue with a single numbering sequence.
Thus, if the questionnaire has three sections, each with 15
items, the first items of the second and third sections would
be numbered "16" and "31," respectively.
When a single item has a number of subparts,
such as with an items-in-a-series format, deciding how to
number the item can be more complex. If the overarching question
is numbered and the subquestions are given no numbers or letters,
this avoids the clutter of an additional set of labels. On
the other hand, not having subquestions labeled makes it difficult
to quickly refer to an item and may create confusion in coding.
Thus, using a set of letters (A, B, C, etc.) for subquestions
may be appropriate.
Minimize the use of directions. If special
directions are needed for a subsection of items or a specific
item, give those directions right before the subsection or
item.
Step 4: For the questionnaire as a whole,
select a presentation and graphic style that maximizes readability
and interest.
Once you have a well-ordered set of items
and a good navigational system, select a presentation style
for the layout of the entire questionnaire. Here, considerations
will differ, depending on whether you are using a paper (hard-copy)
or electronic (computer-based) mode of delivery.
A) The paper questionnaire. If your questionnaire
has more items than can be displayed on one page, consider
how a respondent will read items across multiple pages. In
general, it's wise to use regular letter-sized paper and print
on one side only, being careful to ensure that items do not
break across pages. If the questionnaire is printed on 11"
x 17" paper, it can be folded into a letter-size booklet
whose pages can be turned magazine-style.
If your questionnaire consists mostly of
predetermined-choice items with answers running vertically,
they will be easier to read if they are placed in a two-column
format (a single item running across the entire width of a
regular page is difficult to follow).
It is also helpful to have each item appear
as a discrete block of text, meaning that the spacing between
items should be greater than the spacing within each item.
Other graphic features also affect the appeal
of a questionnaire. If you have made sure the items have a
consistent style and navigational cues (Step 3), make sure
that all the pages also have a consistent layout and style.
For example, every time directions appear, differentiate this
text by size (or style) and, possibly, by boxing it.
Shading also can be used to enhance an entire
set of pages. There is some evidence that more items will
be answered if entire pages are shaded a very light gray (or
other color) with the response boxes left white. Shading must
be light enough so that text is readable.
Step 5: Pilot-test the questionnaire and make
needed revisions.
Once your questionnaire is designed, pilot-test
it with people not participating in the evaluation.
Pilot testers can provide valuable feedback
about how to improve the wording and appearance of the questionnaire.
In the case of an electronic questionnaire, additional pilot
testing is necessary to make sure the questionnaire works
on different computer systems and with respondents who have
varying degrees of computer experience. Although there are
no set rules for how many people should pilot a survey, a
reasonable pilot-test protocol for an average-size evaluation
is as follows:
A) Editing: Have 3-6 individuals who are
familiar with the evaluation (but not involved in the questionnaire
development) read through the questionnaire and make suggestions.
Make revisions.
B) Early Pilot: Give the questionnaire to
4-8 individuals who are similar to your target respondents.
Test each individual separately and have the testers "think
out loud" (i.e., express their thoughts orally) as they
work their way through the questionnaire. Make revisions.
C) Full Pilot: Give the questionnaire again
to 10-15 individuals who are similar to your target respondents.
In this case, try to approximate the conditions you plan to
use for the questionnaire's full implementation. Collect feedback
in the manner most convenient to the situation (e.g., conduct
follow-up interviews and have the testers write down suggestions
and comments). Make revisions.
For larger, higher-stakes evaluations, larger
test groups and more iterations are advised.
How
to design a questionnaire |
Steps
in building a questionnaire |
A
good Questionnaire |
Interviewing
Methods |
Questionnaire
Design
| Ten
Tips to design a Questionnaire |
