Test Items

Test items are the best instrument to measure how well students learned what you intended (learning objectives), as well as how effective instruction was.
Here are some topics about the creation of test items:

• Test score findings
• Appropriate test items
• Well-written test items
• Poorly-written test items
• Templates for patient vignettes

Test Score Findings

The test scores can be translated into two major instructional findings:

• Which students need remediation and
• Which parts of instruction need to be improved

In other words, if few students fail a certain question, these students need remediation but if a great number of students don't perform well, the instruction itself and/or the test item needs to be enhanced.

Appropriate Test Items

The test items that you write must assess the exact performances called for in the learning objectives. The conditions, or givens, in an objective must also be included in the assessment.

A good way to determine whether the desired relationship exists between the objective and test items is to answer the following questions:

• Does the test item require the same performance of the student as that specified in the learning objective?
• Does the test item provide the same conditions or givens as those specified in the learning objective?

Example

Learning Objective:
Given a case of an infant with bronchiolitis, the students will correctly assess the child's physiology and select the appropriate tests to support or confirm the assessment.

Non-Appropriate Assessment Item:
List the most common symptoms of bronchiolitis in an infant.

Appropriate Assessment Item:
Answer the following questions based on the given scenario (infant with bronchiolitis): What do you already know about this child's physiology (air movement and gas exchange)? What laboratory tests should you consider in order to support or confirm your assessment?

Well-Written Test Items

Your test items should clearly indicate the nature of the response that is expected from the student. When your students are unsure about what is expected, their responses may not accurately reflect their knowledge related to an item.

Also, always try to keep your test items free of prompts or clues that students can use to determine correct answers.

• Focus on Important Concept
• Clarity
• Assess Application of Knowledge
• Homogeneous
• Prompt Free
• Interpretation of Data
• Multiple-Choice With Five Choices
• Patient Vignette
• Single Correct Answer

Focus on Important Concept

The question focus on an important concept, typically a common or potentially catastrophic clinical problem.

Clarity

The stem must pose a clear question that has a definite answer. The students should be able to read the question and answer the question without reference to the alternatives.

Assess Application of Knowledge

The question should assess application of knowledge, not recall of an isolated fact (i.e. "two-step" thinking.)

Homogeneous

All distracters (incorrect options) should be homogenous. The use of options that are plausible, even close, but not the best, forces discrimination.

Prompt Free

Questions should not have technical flaws that provide benefit to test-wise examinees, as for example negatively phrased questions and grammatical clues.

Interpretation of Data

When possible, questions should require interpretation of data or information from an image, graph or chart.

Multiple-Choice With Five Choices

Questions should be multiple-choice with at least five choices but if the choices are a list of drugs or structures or enzymes or diseases, the list can be up to ten choices.

Patient Vignette

Whenever possible, a question should use a patient vignette in the stem of the question. Take a look at some templates from the National Board of Medical Examiners (NBME)'s "Constructing Written Test Questions for the Basic and Clinical Sciences" manual.

Single Correct Answer

Single correct answer should be indicated for all questions.

Poorly-Written Test Items

Following are some common practices that result in poorly-written test items :

• Unnecessarily Complex Questions and/or Directions
• Not Specifying Basis for Sequencing/Ordering
• Not Indicating the Nature of the Desired Description
• Using Absolutes
• Using Implausible Alternatives
• Using Equal Numbers of Items for Matching Items
• Including Grammatical Clues
• Having to Reference to Alternatives
• Not Emphasizing Key Words
• Not Specifying What Best Answer Is
• Negatively Worded Questions
• Lengthy Information in Alternatives
• Using "All of The Above"
• Using Trickery

Unnecessarily Complex Questions and/or Directions

Unnecessarily complexity may be due to the amount of information included in the item, to poor grammar, or both.

Not Specifying Basis for Sequencing/Ordering

The basis for ordering things should be specified in the assessment item. For example, putting planets in a certain order can be done in several ways: by distance from the sun, by size, or by alphabetical order.

Not Indicating the Nature of the Desired Description

Most things can be described in at least two ways: by their physical features or by their functions. Test items should indicate both what is to be described and what is to be included in the description.

Using Absolutes

The use of absolutes (e.g., always, never, only, no) should be avoided. Not only absolute statements are usually incorrect, but also some student will know an exception to the keyed answer and get confused or panic.

Using Implausible Alternatives

When multiple-choice items contain alternatives that are obviously incorrect, students have a greater chance of selecting the correct choice.

Using Equal Numbers of Items for Matching Items

When number of items to be paired in matching items is the same it increases the chance of students to guess correctly on items not previously learned.

Including Grammatical Clues

Articles such as "a" and "an", plural word forms, and gender forms may provide clues to correctly answering without having learned the content.

Having to Reference to Alternatives

The students should be able to read the question and answer the question without reference to the alternatives -- and their answer should be among the alternatives when they get there... In other words, the stem asks a question that has a definite answer.

Not Emphasizing Key Words

Anytime you have a key word that would change the answer if the student misses it, boldface it, CAPITALIZE it, underline it or all of the PREVIOUS.

Not Specifying What Best Answer Is

When best-answer items are used, the qualifier in the stem needs to be emphasized and also you must specify in what way an answer is the "best".

Negatively Worded Questions

Negatively worded stems should be avoided.

Lengthy Information in Alternatives

Include as much information in the stem and as little in the options as possible. For example, if the point of an item is to associate a term with its definition, the preferred format would be to present the definition in the stem and several terms as options rather than to present the term in the stem and several definitions as options.

Using "All of The Above"

Don't use "all of the above." Recognition of one wrong option eliminates "all of the above," and recognition of two right options identifies it as the answer, even if the other options are completely unknown to the student. All of the above can also be unfair, because it means answer (A) is correct, so student anticipates (A), looks down, sees (A), chooses it, then moves on. Gets it wrong, even though knew (A) was the correct answer -- gets answer right but question wrong.

Using Trickery

The use of trickery must be avoided. There is no point in finding out if you can trick the students -- you can.

Templates for Patient Vignettes

Templates can be very helpful when creating questions. The following information about question templates is an excerpt from the National Board of Medical Examiners (NBME)'s "Constructing Written Test Questions for the Basic and Clinical Sciences" manual. You can download the full version of the manual, as a PDF file.

• Item templates
• Patient vignettes components
• Additional templates
• Types of questions
• Sample lead-ins and option lists

Item Templates

The overall structure of an item can be depicted by an item template. You can typically generate many items using the same template.

For example, the following template could be used to generate a series of questions related to gross anatomy:
A (patient description) is unable to (functional disability). Which of the following is most likely to have been injured?

This is a question that could be written using this template:
A 65-year-old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty flexing his leg. Which of the following muscles is most likely to have been injured?

• Gluteus minimus
• Hamstrings
• Iliopsoas
• Obturator internus

Patient Vignettes Components

Many basic science questions can be presented within the context of a patient vignette. The patient vignettes may include some or all of the following components:

• Age, gender (e.g. a 45-year-old man)
• Site of care (e.g. comes to the emergency department)
• Presenting complaint (e.g. because of a headache)
• Duration (e.g. that has continued for 2 days)
• Patient history (with family history ?)
• Physical findings
• +/- Results of diagnostic studies
• +/- Initial treatment, subsequent findings, etc.

Additional Templates

• A (patient description) has a (type of injury and location). Which of the following structures is most likely to be affected?

• A (patient description) has (history findings) and is taking (medications). Which of the following medications is the most likely cause of his (one history, PE or lab finding)

• A (patient description) has (abnormal findings). Which [additional] finding would suggest/suggests a diagnosis of (disease 1) rather than (disease 2)?

• A (patient description) has (symptoms and signs). These observations suggest that the disease is a result of the (absence or presence) of which of the following (enzymes, mechanisms)?

• A (patient description) follows a (specific dietary regime). Which of the following conditions is most likely to occur?

• A (patient description) has (symptoms, signs, or specific disease) and is being treated with (drug or drug class). The drug acts by inhibiting which of the following (functions, processes)?

• A (patient description) has (abnormal findings). Which of the following (positive laboratory results) would be expected?

• (time period) after a (event such as trip or meal with certain foods), a (patient or group description) became ill with (symptoms and signs). Which of the following (organisms, agents) is most likely to be found on analysis of (food )?

• Following (procedure), a (patient description) develops (symptoms and signs). Laboratory findings show (findings). Which of the following is the most likely cause?

• A (patient description) dies of (disease). Which of the following is the most likely finding on autopsy?

• A patient has (symptoms and signs). Which of the following is the most likely explanation for the (findings)?

• A (patient description) has (symptoms and signs). Exposure to which of the (toxic agents) is the most likely cause?

• Which of the following is the most likely mechanism of the therapeutic effect of this (drug class) in patients with (disease)?

• A patient has (abnormal findings), but (normal findings). Which of the following is the most likely diagnosis?

Types of Questions

• Guess my drug
• Guess my toxic exposure
• Guess my diet
• Guess my mood
• Predict physical findings
• Predict lab findings
• Predict sequelae
• Identify underlying cause/diagnosis
• Identify cause of drug responses
• Identify drug to administer

Sample Lead-ins and Option Lists

Which of the following is abnormal?

sites of lesions;
list of nerves;
list of muscles;
list of enzymes;
list of hormones;
types of cells;
list of neurotransmitters;
list of toxins, molecules, vessels, or spinal segments

Which of the following findings is most likely?

laboratory results;
list of additional physical signs;
autopsy results;
results of microscopic examination of fluids, muscle or joint tissue;
DNA analysis results;
serum levels.

Which of the following is the most likely cause?

underlying mechanisms of the disease;
medications that might cause side effects;
drugs or drug classes;
toxic agents;
hemodynamic mechanisms, viruses, or metabolic defects.

Which of the following should be administered?

drugs;
vitamins;
amino acids;
enzymes;
hormones.

Which of the following is defective/deficient/nonfunctioning?

enzymes;
feedback mechanisms;
endocrine structures;
dietary elements;
vitamins.

Copyright © 1996, 1998 National Board of Medical Examiners® (NBME®)
Copyright © 2001, 2002 National Board of Medical Examiners® (NBME®)