MULTIPLE CHOICE QUESTIONS Design and Evaluation

1

MULTIPLE CHOICE QUESTIONS
DESIGN AND EVALUATION
Helen Huntley
Senior English Language Fellow
Ministry of Education and Training Hanoi, Vietnam
2
Workshop Goals

To review the advantages and disadvantages of multiple choice tests
To explore tips and techniques for writing successful multiple choice items
To evaluate and correct multiple choice questions and answers
3
Objective Testing

“Tests in which the learning material is divided into small units, each of which can be assessed by means of a test item with a single correct answer that can be specified in advance.”

Reed, J. (2000). Assessing vocabulary. Cambridge University Press
4
Advantages of Multiple Choice Tests
good discrimination among levels of ability
high reliability
valid indicators of overall language ability
highly practical: convenient to administer
well-established procedures to analyze items
writing ability does not affect performance
cost-effective format - easy scoring and grading
familiarity of students with MCQs
5
Disadvantages of Multiple Choice Tests
tendency to treat words as independent linguistic units
difficulty in construction
field-testing, analysis, refinement required
25% chance of guessing correctly
items may test knowledge of distractors
wrong answer may result from lack of understanding of syntax (English)
permits only limited sampling of total knowledge
tests only recognition knowledge
restricts what can be tested
washback may be harmful
cheating may be facilitated
6
Terminology
Every multiple choice item has a stem, which presents a stimulus and several options or alternatives to choose from.
One of these options, the key, is the correct response. The others serve as distractors
7
Match the Test Items to the Blueprint and Task Specifications
Addresses curriculum objectives in terms of performance (can do)
Outline of the test
Skills to be included
Item types and tasks
Topics to be covered
Elicitation and response formats for items
Number of items in each section
Time allocation
Plan for scoring
8
Using Item Indices to Accept, Discard, or Revise Items
Item facility: the extent to which an item is easy or difficult for the proposed group of test takers
Item discrimination: the extent to which an item differentiates between high- and low-ability test-takers
Distractor efficiency: the extent to which the distractors ‘lure’ a sufficient number of test takers
9
Multiple Choice Tips I

Each test item should be designed to measure a specific objective
The question or task should be clear, testing only one concept
The answer should involve interpretation of the material, not just background knowledge
A context should be provided for the items
The items should not show an insensitive or unfair bias
The number of response options should be the same in each question
Ambiguity in the answers must be avoided; the intended answer should clearly be the only correct one.
The options should all be similar in length and level of difficulty
10
Multiple Choice Tips II
Items at different levels of difficulty should be included: start with the easiest first
Both the stem and options should be stated as simply and directly as possible
Writing illogical or nonsensical distractors should be avoided
Extraneous clues should not influence the answer
The correct answers should appear in all positions (A, B, C, D)
Words should not be repeated in the answers
A negative ‘stem’ should be avoided The questions should not be dependent on previous items
Appropriate punctuation should be used
11
Helpful Resources
ESL dictionaries
ESL online dictionaries
http://www.ldoceonline.com/
Thesaurus
Online thesaurus
Grammar reference texts
High school textbooks
Flesh-Kincaid Readability Grade (MSW)
12
Sample MC Questions
Evaluate the following multiple choice items.

What is the objective of each question?
Identify problem areas (see Multiple Choice Tips)
Correct/improve problem areas
13
MCQ#1
Excuse me, do you know _____?

where is the post office
where the post office is
where post office is

14
MCQ#2
An architect is a person who does not _____.

design automobiles
design buildings
design houses
design offices
15
MCQ#3
My sister _____ as an air hostess.

working
works
she works
job
16
MCQ#4
I usually participate in badminton competitions _____ November.

from
on
at
to
17
MCQ#5
Tuesday and Wednesday come before Thursday but after _____.

Sunday
Saturday
Monday
Friday
none of the above
18
MCQ#6
The answer to the last question is the _____ day of the week.

first
forth
three
fifth
19
MCQ#7
Hilary lives in New York City. _____ works in the hotel business.

her
he
him
she
20
MCQ#8
I’m thirsty. I think I’ll drink an _____.

orange
knife
apple cider
umbrella
21
MCQ#9
While Susan prepared the ____ for lunch, her mother watched television.

kitchen
children
chicken
cartoon
22
MCQ#10
Can you type _____ on a computer?

typing
to type
typed
type
23
MCQ#11
Although the plan has some disadvantages, they are outweighed by the _____.

advantages
negatives
drawbacks
problem
24
MCQ#12
Roberto is lazy. He always takes a siesta _____ the afternoon.

in
during
at
throughout
25
MCQ#13
My eyesight has really been deteriorating lately. I wonder if I need glasses. I think I’d better go to the _____ to have my eyes checked.
pediatrician
dermatologist
optometrist
audiologist
26
MCQ#14
We went to visit the pagodas, _____ fascinating.

Which were beautiful
Which were especially
Which were holy
Which were special
27
MCQ#15
Where did Jane go after the birthday party last night?

Yes, she did.
Because she was tired.
To her friend’s house.
She went home around ten o’clock.
28
MCQ#16
Where did George go after the party last night?

Yes, he did.
Because he was tired.
To Elaine’s place for another party.
Around eleven o’clock.

29
Your Turn

Complete the worksheet to find problems in multiple choice question development.
30
Multiple Choice Editing Questions/Error Recognition

The abrasively action of the wind wears
A B
away softer layers of rock.
C D
31
Multiple Choice Editing Questions/Error Recognition
Researchers have discovered that the
A
application of bright light can sometimes
B
be uses to overcome jetlag.
C D
32
Multiple Choice Editing Questions: Paragraph

Ever since supermarkets first appeared, they have been take over the world. Supermarkets have changed people’s lifestyles, yet and at the same time, changes in people’s lifestyles have encouraged the opening of supermarkets. As a result this, many small stores have been forced out of business. Moreover, some small stores will be able to survive this unfavorable situation.

33
GapFill/Cloze Multiple Choice
In years to come, zoos will not (1) _____ be places where animals are exhibited (2) _____ the public, but repositories where rare species can be saved from extinction (3)_____ captive breeding…
a. also
b. only
c. even
d. ever
a. by
b. near
c. to
d. into
34
Vocabulary Multiple Choice
I disagree with a few of his opinions, but _____ we agree.
deliberately
conclusively
essentially
immensely
35
Vocabulary Multiple Choice
Because fewer people are taking expensive vacations, the tourist industry is in a _____.
choke
grope
grumble
slump
36
Item Writing Practice

Further practice in designing multiple choice test items will be provided in specific language skill assessment sessions.
37
Any Questions?
?
?
?
?