How Is the ETS® NOTE™ Program Updated To Keep Content Current?
ETS is committed to keeping its assessment content up to date and aligned with professionally accepted standards, and assessments are, therefore, reviewed on a regular basis. We involve stakeholders in this process by asking them to nominate teachers and teacher educators to serve on Test Development Standing Committees and National Advisory Committees (NACs). These committees are comprised of practitioners that contribute to the development of the Praxis® tests, the NOTE program and other assessments. Teachers and teacher educators interested in serving on one of these committees can register at www.ets.org/praxiscomm.
The Test Development Standing Committees review the specifications for each assessment annually to determine whether there are circumstances that require revision of the assessment specifications or regeneration of the assessment. The questions that the Standing Committees answer include the following:
- Do the assessment specifications still represent knowledge and/or skills that are important for competent, beginning-level practice?
- Are there areas of knowledge and/or skill that are not included but are critical for competent, beginning-level practice?
- Are there content categories in the assessment specifications that we need to update to reflect current practice?
- Are there changes on the horizon that could impact these specifications in the future, either at the national level or within a particular state?
The results of these specification reviews determine the plan for regeneration of an assessment. ETS researchers, psychometricians and assessment developers analyze the reviewers’ answers and comments and use expert judgment to determine a path forward for each assessment. The group would typically choose one of three options, depending on the answers to the questions provided by the Standing Committee:
- Internal and external experts validate the assessment specifications and no National Advisory Committee (NAC) meeting is necessary; assessment specifications remain unchanged.
- When updates are straightforward, we use in-house content experts to draft updated assessment specifications to improve alignment; external experts then validate the specifications via a virtual meeting. We make these updates when the assessment specifications do not match current professional standards in the field.
- When updates are not straightforward, a NAC is convened to update the assessment specifications to improve alignment, and ETS performs a job analysis confirmation survey. These updates are necessary when assessment specifications do not match current professional standards if they exist, or internal and external experts determine the specifications are outdated.
When a NAC is convened, client states help ETS recruit 12–15 licensed practicing professionals who are close to the beginning of their teaching practice, as well as college faculty who prepare teachers for that specific licensure area. The NAC works closely with ETS assessment specialists to define the content domain and later to develop the assessment blueprint and assessment content specifications. NAC meetings typically take place on the ETS campus in Princeton, N.J.
After the NAC meets to define the assessment domains, a job analysis survey is conducted to confirm these domains. A job survey is conducted for new assessment and for existing assessments that are being significantly revised. The job analysis survey enables relatively large numbers of practitioners from across the country to judge the importance of the knowledge and/or skills defined by the NAC for beginning-level practice. The practitioners' judgments serve as an independent source of validation evidence. Once the results of the survey have been analyzed, the NAC is reconvened to develop the assessment blueprint and assessment specifications.
After the NAC develops the assessment blueprint and assessment specifications, ETS convenes panels of licensed practicing educators and college faculty from user states to conduct standard-setting studies. Each standard-setting panel confirms that the knowledge and/or skills represented in the assessment content specifications are important for entry-level practice and recommends a passing score for each state to consider when setting its own passing score.