Situational Judgment Tests (SJT)
ETS has long been interested in the potential usefulness of SJTs as methods for assessment. Research projects have been conducted in response to a strong need and expressed desire on the part of employers for new tests of noncognitive skills such as leadership, teamwork, time management and emotional intelligence. To this end, ETS has also explored innovations in SJT methodology by investigating the use of novel SJT methods selected for their potential to function beyond existing self-report assessments.
One approach involves the use of single-response SJTs, in which respondents evaluate critical incidents directly rather than selecting among a number of effective and ineffective responses to a given scenario.1 Another approach involves the use of observer-report SJTs, in which supervisors or peers are asked to choose how a target respondent would respond in a situation.2
Both methods are hypothesized to provide improved validity or reduced costs compared to traditional self-report and self-report SJT measures, which can be inadequate or inappropriate. An assessment that supports reliable and valid inferences about noncognitive skills could be highly beneficial across industries and the higher education sectors.
Call Center Studies
The main goal of these studies is to assess the extent to which noncognitive skills (and related competencies) predict the performance of customer service representatives (CSRs). A sample of CSRs from multiple call centers, including a site at Prometric®, completed an online survey containing self-report personality items that had been matched to a competency model based primarily on job analyses and company mission statements.
The assessment included innovative measures of personality such as forced-choice assessments paired with Bayesian Truth Serum methodologies. Use of these innovative assessment methods in call center samples represents an important contribution to both research and the labor market. Papers are currently being prepared on these topics for peer review publication.
Job Readiness Exploration
Understanding how students transition from four-year or community colleges and universities to full-time jobs is an important and under-researched concern for scientists and practitioners interested in how well these institutions prepare students for successful employment. The purpose of this small project was to conduct an exploratory investigation of colleges and universities with existing ties or relationships with large companies.
The goal was to identify companies and universities with whom ETS could partner in order to assess whether students are prepared for success in their jobs after graduation. Identifying universities who send large numbers of graduating students to a single company will allow for greater potential sample sizes, access and standardization to help improve the validity of research studies. The major deliverable for the project was a database listing universities, businesses and relevant contact persons.
Adult Transition Learners
The purpose of the Adult Transition Learners: Relationships between Online Non-Cognitive Skills Feedback and Counseling and Post-Secondary Outcomes in Adult Transition Learners study was to repurpose and pilot an online battery of ETS noncognitive assessments designed to be completed independently by adult transition learners at two urban sites that work with adults preparing for postsecondary settings.
In addition to modifying the language of the items to be appropriate for this population, the objective was to determine the feasibility of assessing all constructs in this sample and to obtain reactions to the survey items, particularly the innovative video-based items. Finally, work was completed to develop score reports to benefit career counselors or college mentors, and adult educators in college readiness programs.
Major activities included developing a noncognitive assessment battery and conducting pilot studies with adult learners at two local organizations in Philadelphia, Graduate! Network and District 1199c, ETS's partners in this study. Research from these studies were presented at AERA in San Francisco, April 2013.
Forced Choice Personality Measurement
ETS has been exploring the use of adaptive forced-choice assessment batteries of personality dimensions. To date, several versions of these assessments have been created and administered. Efforts are currently underway to administer these assessments to different sectors of the labor market.
The measurement approach combines modern psychometric methods3 computing technology, and research findings from the personnel selection and personality domains to create a system that is innovative not only in terms of how personality constructs are being measured (i.e., the psychometric underpinnings of the assessment), but also what aspects of personality should be measured, at what level of generality, and for what purposes (i.e., the content of the assessment).
New Perspectives on Faking
ETS staff members edited and contributed to a book on faking entitled New Perspectives on Faking in Personality Assessments. This publication builds on the base of research conducted at ETS.
To learn more about any of these projects, contact ETS at firstname.lastname@example.org
1 Motowidlo, S., Crook, A., Kell, H., & Naemi, B. (2009). Measuring procedural knowledge more simply with a single-response situational judgment test. Journal of Business and Psychology, 24, pp. 281–288.
2 MacCann, C., Wang, P., Matthews, G., & Roberts, R. D. (2010). Examining self-report versus other reports in a situational judgment test of emotional abilities. Journal for Research in Personality, 44, pp. 673–676.
3 Stark, S., Chernyshenko, O. S., & Drasgow, F. (2005). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise preference model. Applied Psychological Measurement, 29, pp. 184–201.