Build a questionnaire people can answer and researchers can trust
A practical guide to writing Likert items, choosing a response scale, testing validity and reliability, and running a pilot before your main study.
I understood every question in this survey without re-reading it.
Try it. This is a fully labelled, 5-point agreement item, the format most of this guide is about.
1Define the construct first
Every item should trace back to one clearly defined thing you want to measure, such as "teacher burnout" or "trust in online journals". Write a one-sentence definition, then list its dimensions. If you cannot say what is outside the construct, you cannot decide which items belong.
- Review the literature and existing scales. Adapting a validated scale is often better than starting over.
- Name the population and the reading level of your respondents.
- Draft an item pool about two to three times larger than your target final length, spread across each dimension.
2Write good items
Good items are short, concrete and answerable by everyone in your sample. Most measurement error at this stage comes from a few repeatable mistakes.
- Keep items under about 20 words and match the wording to your respondents.
- Give each item a single reference period ("in the past month") when time matters.
- Use reverse-worded items sparingly. They can confuse respondents and create artificial factors. Consider balanced wording only if acquiescence bias is a real concern.
- Put sensitive items after easier ones, and keep the order neutral so early items do not prime later ones.
3Choose the response scale
Match the response format to the question. Agreement suits attitudes, frequency suits behaviours, and importance or satisfaction suit evaluations. Then decide how many points to offer.
| Decision | Guidance |
|---|---|
| Number of points | Five or seven points work well for most studies. Fewer than four loses information, and more than seven rarely adds reliability. |
| Odd or even | An odd scale gives a midpoint for people who are truly neutral. An even scale forces a lean, which can distort answers if some respondents have no opinion. Choose deliberately. |
| Labels | Label every point, not just the ends. Fully labelled scales are easier to interpret consistently. |
| Direction | Keep the same direction (negative to positive or the reverse) across the whole questionnaire. |
| "Don't know" | Add "not applicable" as a separate option, outside the scale, when some respondents may lack the experience. |
Analysis note: a single Likert item is ordinal. A summed or averaged score from a well-built multi-item scale is commonly treated as interval-like, but report medians or use ordinal methods when you analyse individual items.
4Test content validity
Content validity asks whether the items cover the construct fully and only the construct. Ask a panel of experts to rate each item's relevance on a 4-point scale (1 = not relevant, 4 = highly relevant).
- Panel: aim for 5 to 10 experts with subject or measurement expertise. Add a few members of the target population for wording feedback.
- Item CVI (I-CVI): the share of experts rating the item 3 or 4. With six or more experts, keep items at about 0.78 or higher. With five or fewer experts, aim for full agreement.
- Scale CVI (S-CVI/Ave): the average of the I-CVIs. Aim for about 0.90 or higher.
- Face validity: run short cognitive interviews. Ask respondents to think aloud and say the item in their own words.
Item CVI calculator
5Run a pilot
A pilot finds problems while they are cheap to fix. Run it in two rounds and keep the conditions close to the real study.
Round A: cognitive interviews (5 to 15 people)
Watch for hesitation, re-reading, requests for clarification and answers that show the item was understood differently than you intended. Rewrite before moving on.
Round B: pilot survey (about 30 to 100 respondents)
- Use the same mode, platform and instructions as the final study.
- Record completion time, drop-off points and skipped items.
- Check the response distribution. Items where nearly everyone chooses the same option add little information.
- Look at item-level missing data and any comments left in an open box.
- Keep pilot participants out of your final sample.
Decide beforehand what will make you drop or rewrite an item, such as a corrected item-total correlation below 0.30 or a ceiling effect, so decisions are not made after seeing the results you hope for.
6Test reliability
Reliability is consistency. Check internal consistency in your pilot, and stability over time if your construct should be stable.
| Method | What it shows | Common benchmark |
|---|---|---|
| Cronbach's alpha | How well items in one dimension hang together | 0.70 acceptable, 0.80 or more good. Above about 0.95 may signal redundant items. |
| McDonald's omega | A model-based alternative that does not assume equal item loadings | Similar cut-offs to alpha |
| Corrected item-total correlation | Whether each item tracks the rest of its scale | 0.30 or higher |
| Test-retest (ICC) | Stability across two administrations, typically one to two weeks apart | 0.70 or higher |
Compute alpha separately for each dimension, not across a multidimensional questionnaire. Alpha rises with the number of items, so a high value on a long scale is not proof of quality.
See how alpha responds to length
Uses the standardized formula α = k·r̄ / (1 + (k−1)·r̄). Real alpha from raw data will differ.
7Test construct validity
Once the pilot has cleaned up obvious problems, collect a larger sample to test the structure of the scale.
- Sample size: plan for at least 5 to 10 respondents per item and not fewer than 100 to 200 overall. Larger samples are safer.
- Exploratory factor analysis (EFA): use it when the structure is unknown. Check KMO (0.60 or higher) and Bartlett's test, then retain items loading about 0.40 or higher on their intended factor without strong cross-loadings.
- Confirmatory factor analysis (CFA): use a new sample to test the structure. Common fit targets are CFI and TLI near 0.95 or above and RMSEA near 0.06 or below.
- Convergent and discriminant validity: your scale should correlate with measures of similar constructs and less with unrelated ones.
Do not run EFA and CFA on the same data. Split the sample or collect a second one.
8Before you launch
0 of 8 done