New constructs deal with learner characteristics not adequately represented in ETS's current large-scale testing programs. Work in this area includes:
- identifying new constructs
- developing new measures
- exploring new uses
- pursuing new opportunities
The application of new constructs isn't limited to educational admissions; ETS also develops them for:
- teacher professional development
The division also pursues applications for industry, military and international markets.
Identifying New Constructs
ETS conducts ongoing reviews to identify current research on viable new constructs. Among the key construct features considered are validity, modifiability and amenability to assessment. The review serves several purposes.
- It expresses current knowledge of the status of various constructs: what ETS currently measures; where there's research evidence for potential incremental validity; what might be modified; and what might be assessed as part of a richer outcome assessment scheme.
- It communicates ETS knowledge to potential clients looking for innovative assessments and new uses such as outcome assessment. It highlights constructs that broaden the understanding of group differences and provides direction for fairness and equity initiatives.
- It flags expansion opportunities where there is a good research base for developing an assessment product, but where no such product currently exists.
In the testing arena, there's a need to expand the variety of characteristics assessed. These range from realistic, simulation-like assessments (e.g., problem-solving in context), to qualities such as teamwork and emotions that have been largely ignored in testing high-stakes outcomes. ETS is also interested in expanding the range of outcome measures for assessing interventions — such as attitudes, personality, emotion and motivation.
Developing New Measures
While identifying new constructs, ETS simultaneously explores practical ways to assess them. In the last few years, several new measures were developed in response to client needs. They include:
- Test of Listening Communication Skills. This interactive multimedia test presents real communication situations on video with actors. Examinees evaluate various ways to deal with the communication situations.
- Standardized Interview Procedure. This is an extension of the Standardized Letter of Recommendation. Interviews are often used in industry and in clinical specialties to elicit soft qualities — such as motivation and people skills — which can't be easily measured with existing tests.
- Performance Measures of Personality. Such measures are meant to supplement personality self-assessments, which are widely used in industry, but also are of questionable validity.
- Survey of Teacher Attitudes. The survey helps teachers become aware of attitudes that may facilitate or hinder the learning environment within their classrooms. Attitudes can also affect teaching practices. ETS works with different school districts to identify teacher attitudes, and to develop interventions that ensure attitudes are productive ones.
- Survey of Cultural Attitudes, Values and Beliefs. This survey captures the key psychological dimensions on which cultures vary. The survey explores differences between cultures in an effort to facilitate cross-cultural understanding and promote cultural sensitivity.
- Web-Based, Problem-Solving Skills Assessment. This assessment is currently being used by a major international strategic consulting firm to screen business analysts. It features situational judgment test items that satisfy the screening purpose and also serve as job previews.
A key issue in the development of new measures is eliciting expert knowledge using cognitive task analysis.
Other issues include the
- use of novel methodologies, such as the "situational judgment paradigm," automatic item generation from specifications, and analysis of whether to use off-the-shelf assessments or develop new ones.
- scoring of new construct measures. This often represents a significant challenge: while many options are available in the literature for new constructs assessment, none has yet been accepted as the gold standard.
Exploring New Uses
Self-help, self-awareness, diagnosis, policy and guidance are all functions that assessments can play beyond their traditional role in admissions or selection decisions. This effort explores such new uses with assessments currently in development. Much of this work is done with broad factors such as personality, interests and attitudes.
Teaching Success and Retention
ETS staff believes that interests, personality and attitudes can play an important role in teaching success and retention. In development are strategies to guide prospective teachers into areas that best match their interests and personality. ETS believes this will lead teachers toward greater job satisfaction and better performance.
Metacognition in Teaching and Learning
Metacognition pertains to knowledge about cognition and cognition regulation — with planning, self-monitoring and self-evaluation being the most important components of the latter.
Metacognition is important for the overall regulation of cognitive activity and, through its links with constructs of self-efficacy and self-concept, has an important role as a motivational force. ETS research staff examines the role of metacognition in school-related activities from the teacher and learner points of view. They also examine life-span changes in components of metacognition, such as self-confidence and self-monitoring.
Pursuing New Opportunities
ETS's tests and research have historically been applied to educational institutions in the United States. But ETS also seeks to be a leader in providing "products and services wherever people or institutions are held accountable for learning." This means pursuing new opportunities in government, military, industrial, medical and international arenas.
R&D staff has experience doing research for the military and other agencies that might benefit from the power of both cognitive and socioaffective measures. Industry and foreign education officials have articulated the importance of new constructs in education and the workplace. Already developed new constructs may be adapted for these uses — and for screening and outcome assessment purposes. The military has expressed interest in issues related to peacekeeping and other international missions, including awareness of cultural differences; and assessment and development of emotional competence and conflict resolution skills.
Emotional Intelligence and Competencies
This project aims to
- develop a suite of emotional intelligence measures
- derive interventions targeted at improving social and emotional competencies
- apply measures and interventions in education, applied psychology (especially organizational and military) and gerontology.
Situational Judgment Test Items
The purpose of the effort is to explore the feasibility of situational judgment tests for various constructs, such as communications, teamwork and critical thinking.
Games and Simulations as Assessments
This project explores the use of games and simulations as assessment tools. The approach will pursue two aims simultaneously:
- a review of the games and simulation literature that will focus on how knowledge, skills and abilities are typically assessed in the game/simulation context and that will conclude with strategies for improving simulation assessments
- prototype game/simulation environments to assess various new constructs, such as learning ability; judgment and decision making; adaptability; emotional intelligence; metacognition; leadership; and attitudes, values and beliefs