The Criterion Problem: What Measure of Success in Graduate Education?
- Hartnett, Rodney T.; Willingham, Warren W.
- Publication Year:
- Report Number:
- ETS GRE Board Report
- Document Type:
- Page Count:
- Subject/Key Words:
- Evaluations Indicators of Success Performance Measures Selection Measures Student Performance
A wide variety of potential indicators of graduate student performance are reviewed. Based on a scrutiny of relevant research literature, experience with recent and current research projects, and conversations with graduate faculty members and administrators, the various indicators are considered in two ways. First, they are analyzed within the framework of the traditional "criterion problem," that is, with respect to their adequacy as criteria in predicting graduate school performance. In this case emphasis is given to problems with the criteria that make it difficult to draw valid inferences about the relationship between selection measures and performance measures. Second, the various indicators are considered as an important process of the graduate program. In this case, attention is given to their adequacy as procedures for the evaluation of student performance--e.g., their clarity, fairness, and usefulness as feedback to students. The assessment of graduate student performance is complex, both conceptually and technically, and the overall system of evaluation seems reasonably fair and sound in the sense that the students earning degrees are undoubtedly those who are more competent and deserving. Viewed separately, however, each of the various indicators of success has numerous shortcomings. As a result, the measures normally available for validation studies, for evaluation of student performance, or for program evaluation are often not as good as they should or could be. Various general observations about the status of student evaluation practices are made, and particular attention is drawn to the view that many evaluation practices seem to be characterized by ambiguity with regard to their basic purpose. Finally, suggestions are offered for how assessment practices might be improved.