Writing good questions that really test learners’ knowledge takes time and careful consideration. Good questions should cover relevant content and be well-structured, and should not be difficult to interpret. The questions should test knowledge and skills, and not just recall of information, and should not include cues that give learners who are familiar with taking tests an advantage, i.e. learners should not be able to work out the correct answer, and eliminate the incorrect answers based on the format of the question.
Tests also need to be reliable, valid and fair. A reliable test produces consistent results, irrelevant of when or where it takes place; and a valid test measures what it is supposed to measure. A test can be reliable without being valid. A test should not favour certain population groups over others, i.e. they should be fair and without bias.
An exam is a ‘high stakes’ test, i.e. it is important that it is set up correctly because the consequences of not doing so can have a significant impact on learners.
Firstly, though, you need to know what the learning outcomes for your module are! Then give some thought to the following:
- What topics do you want to cover?
- What are the essentials that all learners should know?
- What are the typical misconceptions in this subject area?
- What questions do you want to ask?
Multiple Choice Questions (MCQs)
MCQs are used to test knowledge objectively. They can be structured to test higher order skills such as understanding, application of knowledge and evaluation of information.
MCQs consist of a lead-in question or statement, commonly known as ‘the stem’, followed by a list of options (usually five), and an instruction to select one answer. Basic questions have one correct answer. More advanced questions require learners to select the best match answer from a list of potentially correct answers, called a Single Best Answer type.
Two typical problems to avoid are:
- Writing questions that allow learners to guess the answers.
- Writing questions that confuse learners into picking the wrong answer.
To start with, take a look at Cathy Moore’s quiz which illustrates some common mistakes to avoid: Can you answer these 6 questions about multiple-choice questions?
Phil Race also identifies some common mistakes in designing multiple-choice questions. Visit the Phil Race: Archived downloads webpage and download the zip file under ‘Designing Multiple-Choice Questions’ and view the from slides from 12 onwards). In summary:
- Don’t include a word from the stem of the question in one of the options, or it will make people think it is the correct option.
- An option can look more probable if it is longer and because it is qualified.
- Don’t mix indefinites with definites: indefinites are more likely to be correct (e.g. usually, sometimes, often, seldom); definites are not (e.g. all, never, none).
- Don’t allow the indefinite article (‘an’) to give away the right answer.
- Don’t ask for something plural, then make just one option plural.
- Be careful about overlapping questions.
- Be careful not to use a pattern of responses.
Creating Good Self-Test Questions
From Leonie Sloman’s Creating Good Self-Test Questions resource, some more useful information below.
Learners can make an intelligent guess sometimes by:
- Choosing the longest answer.
- Choosing the most carefully-worded answer, including qualifications such as ‘generally’ or ‘usually’, and avoiding absolutes such as ‘always’, ‘never’.
- Choosing ‘all of the above’ or ‘none of the above’.
- Looking for grammatical cues, e.g. a question ending with ‘an’ suggests that the correct answer starts with a vowel.
- Looking for repeated words or information − choosing the answer with the most elements in common with the other answers.
Avoid confusing questions:
- Don’t try to cover too much in one question.
- Avoid long, complicated answers.
- Avoid negatives, or make them very clear, e.g. ‘Which of the following is NOT an example of X.’
- Be consistent in how you express numbers, including proportions (i.e. half or 50%).
- Be careful how you express frequency. People can interpret ‘usually’, ‘often’, ‘occasionally’, ‘likely’, ‘commonly’, ‘rarely’ etc. in very different ways.
Other general advice:
- Put as much of the text in the question stem as possible, i.e. try to keep the answers short. Ideally, after reading the question stem, the learner would have an idea of what answer they are looking for before seeing the options.
- If some options are partially correct, ask learners to select the best
- Include at least two distractors (incorrect answers) within your list of answers.
- Ensure all the distractors are plausible and a similar style and length to the correct answer.
- Good distractors might be common misconceptions or correct facts that aren’t relevant to the question asked.
- Most software allows the order of the answers to be randomised. However, if your answers can be ordered by size, it’s more logical to list them from smallest to largest, e.g. a) 1-5, b) 6-10, c) 11-15, d)16-20.
Evaluating your MCQ test
After the questions have been written, carry out a practice run on a group of learners with similar characteristics to those who will take the exam. This will enable you to check the following:
- The difficulty of each question.
- Whether the questions are ambiguous or misleading.
- Whether revisions are needed to the questions.
- Whether some questions should be eliminated.
- Whether any answers should be revised, replaced or removed.
- That the content of the questions and answers is inclusive, i.e. that it isn’t sexist, racist, inappropriate or offensive to anyone.