Assessing Vocabulary

Assessment of Vocabulary

Helen Huntley
Senior English Language Fellow 2007-08
MOET, Vietnam
[email protected]
http://helenshuntley.com/


Workshop Explorations
What is a word?
Does vocabulary consist of single words or as larger lexical items?
What does it mean to know a lexical item?
What should be measured in a vocabulary test?
How should vocabulary be measured?
What is better – discrete-item testing or contextualized-item testing?
And more…
Questions to Ponder
How is vocabulary tested in the school graduation and university entrance tests?
How are the words chosen for inclusion in the test?
What features of a word are being tested?
What is a word?
Single words: chair
Phrasal verbs: put on
Short phrases: as a matter of fact
Longer phrases: it has come to my attention that…
idioms: out to lunch
Larger lexical items
Phrasal verbs
move out, put up with, put off
Compound nouns
personal computer, Applied Linguistics
Idioms
let the cat out of the bag, raining cats and dogs
Collocations
a piercing scream, analyze data
Lexical phrases
Lexical phrases
Polywords: short fixed phrases
at any rate, so to speak, as a matter of fact
Institutionalized expressions: proverbs, formulaic utterances
once upon a time, long time no see
Semi-fixed expressions: basic frame + options
a day/month/year ago, yours sincerely/truly, as far as I know/can tell/ am aware
Sentence builders: framework for complete sentence with slots for whole ideas
I think that…, not only… but also…, I regret to inform you that…
More lexical phrases
I thought you’d never ask.
Call me as soon as you get home.
It’s on the tip of my tongue.
Have you heard the news?
Never say never.
I’ll believe it when I see it.
She never has a bad word to say about anyone


Characteristics of lexical sequences
Relatively fixed in form
The meanings of the individual words do not provide the meaning for the phrase
Familiar expressions for everyday communication
Pragmatic function
Components of word knowledge I: Form: (Nation, 1990: 31)
Spoken Form
(R) What does the word sound like?
(P) How is the word pronounced?
Written Form
(R) What does the word look like?
(P) How is the word written and spelled?
Components of word knowledge II: Position (Nation, 1990: 31)
Grammatical Patterns
(R) In what patterns does the word occur?
(P) In what patterns must we use the word?
Collocations
(R) What words or types of words can be expected before or after the word?
(P) What words or types of words must we use with this word?


Components of word knowledge III: Function (Nation, 1990: 31)
Frequency
(R) How common is the word?
(P) how often should the word be used?
Appropriateness
(R) where would we expect to meet this word?
(P) Where can this word be used?
Components of word knowledge IV: Meaning (Nation, 1990: 31)
Concept
(R) What does the word mean?
(P) What word should be used to express this meaning?
Associations
(R) What other words does this word make us think of?
(P) What other words does this word make us use instead of this one?
Four stages of knowing a word
Stage 1: I never saw it before.
Stage 2: I’ve heard of it, but I don’t know what it means.
Stage 3: I recognize it in context – it has something to do with…
Stage 4: I know it.
(Stage 5: I can distinguish this word from others that are closely related to it.)
Dale (1965: 898)
What is vocabulary ability?
The context of vocabulary use
generational usage
language variations
everyday usage/specialized terminology
Vocabulary knowledge and fundamental processes
vocabulary size
knowledge of word characteristics
lexical storage
fundamental vocabulary processes
Metacognitive stategies for vocabulary use
strategic competence (paraphrazing, avoiding, language switching, appealing to authority, using general terms)

Objective testing of vocabulary
“Tests in which the learning material is divided into small units, each of which can be assessed by means of a test item with a single correct answer that can be specified in advance.”
Reed, J. (2000). Assessing vocabulary. Cambridge University Press
Words treated as independent linguistic units > multiple-choice, matching test, discrete item testing etc.
Tests often based on word lists of frequency
Multiple-choice tests discriminate well among levels of ability and are highly reliable
Valid indicators of overall language ability


Multiple-choice vocabulary items
Little ongoing research available, except for TOEFL

Difficult to construct; require field-testing, analysis, refinement
Learner may know an alternate meaning of the word, but not the one sought
25% chance of guessing correctly
Items may test knowledge of distractors instead of exact meaning of target word
Wrong answer may result from lack of understanding of syntax
Permits only limited sampling of total vocabulary knowledge

Weshe & Paribakht (1996: 17).

BUT: convenient to administer; well-established procedures to analyze them


Matching items
Next to each word, write the letter of its meaning.

Matching items
Next to each word, write the number of its meaning.

Matching format
Find the word which fits in each sentence and write it in the blank at the end.
Someone who is not worried about life is _____.
A metal that is not processed is _____
An illness that never gets better is _____
Clothes that fit close to your body are _____
Someone who hits other people hard is _____

violent
secure
crude
hostile
tight
chronic
parallel
Blank-filling/completion
Write one suitable word in each blank.
Modern planes fly at an a__________ of 35,000 feet.
Many diamond mines are l_________ in South Africa.
Near the Equator, there is not much seasonal v_________ in temperature.
Multiple-choice matching

Last week while Jack and Linda were having lunch in a café, thieves (1)broke down the front door of their house, went inside, and (2)broke into their safe. Now they’re (3)broke.

_____ a. poor
_____ b. made into two or more pieces
_____ c. entered to steal something
_____ d. entered by force


Assessing quality of vocabulary knowledge
Little research available on in-depth study of word-knowledge, except in young children
BUT: proposals to recognize
Partial vs. precise knowledge
Depth of knowledge
Receptive vs. productive knowledge

Sentence-writing
Write a sentence to show you know what the word means and how it is used.
vegetation
involve
twist
starve
principal

Testing depth of processing
TO INTERPRET
Write two sentences: A and B. In each sentence, use the two words given.
interpret experiment
interpret language
2. Write three words than can fit in the blank.
to interpret a(n) i __________
ii __________
iii __________
3. Write the correct ending for each word:
a. Someone who interprets is an interpret____.
b. Something that can be interpreted is interpret_____.
c. Someone who interprets gives an interpret_____.

Cloze tests
Used for assessment of overall language proficiency, reading, and vocabulary
Fixed ratio (every 7th word)
Selective-deletion (rational)
Multiple-choice
C-test
How to separate out vocabulary from other skills?




The word-associates test
Circle the word(s) which are associated with the underlined word.
Edit
arithmetic film pole publishing
revise risk surface text

Assessing vocabulary size
Native-speaker
Anderson (1996): the average 12th grader knows about 80,000 words in L1; children learn up to 4,000-6,000 new words each year

Miller (1977): children between 6-8 pick up an average of 14 new words a day

Nagy & Herman (1987): children acquire 3,000 words per year between grades 3-12

Non-Native Speaker
Satarsyah, Nation & Kennedy (1994): knowledge of 4000-5000 words necessary to understand an undergraduate economics textbook

Nation (1990): a vocabulary of at least 3,000 word families is necessary to read an academic text with 95% comprehension

Nurweni & Read (1999): first-year university students in Indonesia (w/ 6 years of English) had a vocabulary of about 1,200 words.
How to measure vocabulary size?
Wide variation: students in U.S. universities know 15,000 - 200,000 words!
Issues
What counts as a word?
How do we choose which words to test?
How do we find out whether the selected words are known?
The Vocabulary Levels Test
(Nation: 1983; 1990)
Test design
5 levels of word frequency: 2,000, 3,000, 5,000, University word level (5,000+), 10,000.
Word-definition matching (matching
the words to the definition)
Each level has 36 words and 18 definitions
Designed to involve as little reading as possible and to minimize chances of guessing correctly
Intended as a broad measure of word knowledge
Words selected from frequency list on a random basis
Often used as an informal diagnostic tool by teachers; little research done
The Vocabulary Levels Test

Productive Levels Test
I’m glad we had this opp______ to talk.
He takes cr______ and sugar in his coffee.
The telegram was deli______ two hours after it had been sent.
Suddenly he was thru______ into the dark room.
The victim’s shirt was satu______ with blood.
The Eurocenters Vocabulary Size Test I
Test design
Makes an estimate of vocabulary size using a graded sample of words covering numerous frequency levels
Checklist test – learner indicates whether they know the word or not
Includes non-words
Administered by computer (10 minutes)
Instant scores
Initially developed as a placement test
Points subtracted for ‘yes’ answers to non-words
Problematic with low levels of English and French speakers
The Eurocenters Vocabulary Size Test II
The Vocabulary Knowledge Scale
Test design
Can be used with any set of words
Learners are asked to rank words according to 5 scales
Used as a measure of incidental vocabulary acquisition
Designed to measure depth of vocabulary knowledge
The Vocabulary Knowledge Elicitation Scale
Self-report categories

I don’t remember having seen this word before.
I have seen this word before, but I don’t know what it means.
I have seen this word before, and I think it means _____ (synonym or translation)
I know this word. It means _____ (synonym or translation)
I can use this word in a sentence: _____.
(Write a sentence.) (If you do this section, please also do Secton IV.)
(Paribakht & Wesche, 1997: 180)
The Test of English as a Foreign Language (TOEFL)
1996-mid 1970s: separate vocabulary test; discrete-point approach (synonym matching and sentence completion)
1976: words in context vocabulary items included in Reading section
1995: separate set of vocabulary items eliminated; integrated and contextualized in the reading section
1998: computer test; clicking on synonyms within reading
2005: iBT TOEFL; clicking on synonyms within reading; assessing vocabulary in writing and speaking sections
TOEFL: until the mid-1970s:

A _____ is used to eat with.

plow
fork
hammer
needle
foolish


clever
mild
silly
frank
TOEFL: 1976
He discovered a new route through the mountains.
wanted
found
traveled
captured
Their success came about as a result of your assistance.
according to
before
because of
during
TOEFL: 1995
Many of the computing patterns used today in elementary arithmetic, such as those for performing long multiplications and divisions, were developed as late as the fifteenth century. Two reasons are usually advanced to account for this tardy development, namely, the mental difficulties and the physical difficulties encountered in such work.
The word ‘tardy’ in line 3 is closest in meaning to
historical
basic
unusual
late
TOEFL: 1998
The Southwest has always been a dry country, where water is scarce, but the Hopi and Zuni were able to bring water from streams to their fields and gardens through irrigation ditches. Because it is so rare, yet so important, water played a major role in their religion.
Look at the word rare in the passage. Click on the word in the bold text that has the same meaning.
iBT TOEFL
The expenditure of energy is essentially a process of budgeting, just as finances are budgeted. If all of one’s money is spent on clothes, there may be none left to buy food or go to the movies. Similarly, a plant or animal cannot squander all its energy on growing a big body if none would be left over for reproduction, for this is the surest way to extinction.

The word squander in the passage is closest in meaning to
extend
transform
activate
waste
VOCABULARY MULTIPLE CHOICE ITEMS: PROBLEM AREAS

Evaluate the following multiple choice items for vocabulary.
Identify problem areas
Correct/improve problem areas

#1
1. She gets on well with her neighbors __________.
A. at moment
B. lately
C. recently
D. currently
#2

2. That old man is extremely __________.
A. generous
B. strict
C. difficult
D. anxious
#3
3. She designs very beautiful clothes. She is a famous fashion __________.
A. tailor
B. designer
C. producer
D. maker
#4

4. He __________ quickly from his illness.
A. came over
B. got rid of
C. recovered
D. took off
#5
5. Vietnam has been launching a campaign for hunger __________ and poverty reduction.
A. elimination
B. eradication
C. omission
D. decrease
#6
6. The __________ of penicillin has opened a new era for medicine.
A. invention
B. inspection
C. discovery
D. combination
#7
7. The children’s __________ to their parents made them angry.
A. react
B. reacting
C. reaction
D. reacted
Application I
Partner Activity
30 minutes

Write AT LEAST 5 multiple choice vocabulary questions from the vocabulary items provided

Application 2
Partner Activity
30 minutes

Complete a gap-fill vocabulary test for each text provided

Any Questions?
? ? ? ?
? ? ? ?
? ? ? ?
?? ?