skip to main content skip to footer

Creating Vocabulary Item Types That Measure Students' Depth of Semantic Knowledge NLP

Deane, Paul; Lawless, Rene; Li, Chen; Sabatini, John P.; Bejar, Isaac I.; O'Reilly, Tenaha
Publication Year:
Report Number:
ETS Research Report
Document Type:
Page Count:
Subject/Key Words:
Vocabulary Tests, Item Types, Word Knowledge, Topical Associations, Usage Patterns, Conceptual Relationships, Automated Scoring and Natural Language Processing


We expect that word knowledge accumulates gradually. This article draws on earlier approaches to assessing depth, but focuses on one dimension: richness of semantic knowledge. We present results from a study in which three distinct item types were developed at three levels of depth: knowledge of common usage patterns, knowledge of broad topical associations, and knowledge of specific conceptual relationships. We attempted to avoid common sources of variance across items (such as attractive distracters) and hypothesized that the item types that required greater depth of semantic knowledge would tend to show greater difficulty and discrimination after other sources of variance were accounted for. Our results, while still exploratory, support the conclusion that the item types measure different aspects of lexical knowledge, consistent with the hypothesis of increasing semantic depth.

Read More