Research Publications from the CBAL™ Initiative
Preliminary Reading Literacy Assessment Framework: Foundation and Rationale for Assessment and System Design
J. Sabatini, T. O’Reilly, P. Deane
ETS Research Report No. RR-13-30
This report describes the foundation for an assessment framework designed to measure reading literacy based on models of cognitive development. The framework described in this paper was informed by the CBAL™ English Language Arts Competency Model and Provisional Learning Progressions. View the abstract or download this report.
A CBAL™ Science Model of Cognition: Developing a Competency Model and Learning Progressions to Support Assessment Development
L. Liu, A. Rogat, M. Bertling (2013)
ETS Research Report No. RR-13-29
This report describes the CBAL science competency model and three of its related learning progressions for middle school science. Some examples from a formative assessment prototype are presented to illustrate how the competency model can be used to inform assessment development. View the abstract or download this report.
Automated Scoring of Mathematics Tasks in the Common Core Era: Enhancements to M-rater in Support of CBAL Mathematics and the Common Core Assessments
J. Fife (2013)
ETS Research Report No. RR-13-26
This report describes recent enhancements to the ETS m-rater scoring engine to support automated scoring for innovative CBAL mathematics tasks that presented new challenges. View the abstract or download this report.
Exploring Teachers' Understanding of Graphical Representations of Group Performance
D. Zapata-Rivera, M. Vezzu, W. VanWinkle
ETS Research Memorandum No. RM-13-04
In this paper we report on the results of a study that explored the effectiveness of three graphical representations of score distributions aimed at helping teachers understand and effectively use assessment results. View the abstract or download this report.
Automated Essay Scoring in Innovative Assessments of Writing from Sources
P. Deane, F. Williams, V. Weng, C. S. Trapani (2013)
Journal of Writing Assessment, Vol. 6, No. 1.
In this study, data from two large administrations of CBAL™ scenario-based writing assessments were evaluated using automated scoring. The results show that automated scoring can contribute accurate information about writing proficiency when used in combination with other sources of evidence. View the full article on the publisher's website.
Dimensionality Analysis of CBAL Writing Tests
J. Fu, S. Chung, M. Wise (2013)
ETS Research Report No. RR-13-10
This report presents the results of a study to investigate the dimensionality of four CBAL writing tests using exploratory and confirmatory factor analyses. The tests were administered to grade 8 students in fall 2009, and the results show that the tests were multidimensional and support subscore structures and bifactor models. View the full abstract or download this report.
Using Argumentation Learning Progressions to Support Teaching and Assessments of English Language Arts
Y. Song, P. Deane, E. A. Graf, P. van Rijn (2013)
ETS R&D Connections, No. 22
This report explains the CBAL argumentation learning progressions and shows how assessment tasks developed around the learning progressions can be used to support classroom instruction. View the full abstract or download this report.
On the Relation Between Automated Essay Scoring and Modern Views of the Writing Construct
P. Deane (2013)
Assessing Writing, Vol. 18, No. 1, pp. 7–24
This paper explores which aspects of the writing construct are currently measured directly by the kinds of features used in automated essay scoring models. It also examines the implications for validity and for different use cases for automated essay scoring, using the CBAL competency model as a framework for evaluation. View the full abstract or order this article from the publisher.
Covering the construct: An approach to automated essay scoring motivated by a socio-cognitive framework for defining literacy skills
P. Deane (2013)
Chapter in M. D. Shermis & J. Burstein (Editors), The Handbook of Automated Essay Evaluation: Current Applications and New Directions. Routledge.
This paper presents the CBAL understanding of the writing construct and explores which aspects of writing are currently measured directly by the kinds of features used in automated essay scoring models. They conclude that AES can usefully be combined with other measures of writing, provided that one understands that AES primarily characterizes the ability to produce grammatical, coherent, well-developed multiple-paragraph texts, and does not address deeper aspects such as quality of argumentation. Learn more about the book on the publisher's website.
Statistical Report of Fall 2009 CBAL Writing Tests
J. Fu, S. Chung, M. Wise (2013)
ETS Research Report No. RR-13-01
This report presents the statistical results from the administration of four CBAL writing assessments to grade 8 students in twelve states in the fall of 2009. View the full abstract or download this report.
Statistical Report of 2011 CBAL™ Multistate Administration of Reading and Writing Tests
J. Fu and M. Wise (2012)
ETS Research Report No. RR-12-24
This report presents the statistical results from the administration of CBAL reading assessments to grade 7 students and writing assessments to grade 8 students in twenty states in spring 2011. View the full abstract or download this report.
A Preliminary Analysis of Keystroke Log Data from a Timed Writing Task
R. Almond, P. Deane, T. Quinlan, and M. Wagner (2012)
ETS Research Report No. RR-12-23.
This report documents the approaches used to capture the keystroke logs and the algorithms used to process the outputs from CBAL™ Writing assessment pilot tests. In the pilot data, many of the features extracted from the keystroke logs were correlated with human scores. View the full abstract or download this report.
Applying Score Design Principles in the Design of Score Reports for CBAL™ Teachers
D. Zapata-Rivera, W. VanWinkle, and R. Zwick (2012)
ETS Research Memorandum No. RM-12-20.
This paper describes a framework for developing and evaluating score reports, presents several web-based score report prototypes, and summarizes the results of two studies designed to evaluate the score reports with teachers. View the full abstract or download this report.
The Case for Scenario-Based Assessments of Reading Competency
K. M. Sheehan and T. O'Reilly (2012)
Chapter in J. Sabatini, E. Albro, & T. O'Reilly (Editors), Reaching an Understanding: Innovation in How We View Reading Assessment. R&L Education.
This book represents initial attempts to apply theory in order to guide development of new assessments and measurement models. Learn more about the book on the publisher's website.
Rethinking K–12 Writing Assessment to Support Best Instructional Practices
P. Deane, J. Sabatini, and M. Fowles (2012)
Chapter in C. Bazerman, C. Dean, J. Early, K. Lunsford, S. Null, P. Rogers, & A. Stansell (Editors), International Advances in Writing Research: Cultures, Places, Measures. Parlor Press.
This chapter presents some preliminary CBAL™ research results specific to writing. Learn more about this book at the publisher's website.
Exploring the Role of Games in Educational Assessment
D. Zapata-Rivera and M. Bauer (2012)
Chapter in M.C. Mayrath, J. Clarke-Midura, & D. Robinson (Editors), Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research. Information Age.
This chapter describes how some aspects from gaming technology may have a positive effect in the development of innovative assessment systems. Several examples of educational games and game-like assessment tasks including a game-based score report for students developed as part of the CBAL initiative are used to illustrate some of these concepts. Learn more about the book on the publisher's website.
Technology Enhanced Assessments in Mathematics and Beyond: Strengths, Challenges, and Future Directions
G. Cayton-Hodges, L. Marquez, P. van Rijn, M. Keehner, C. Laitusis, D. Zapata-Rivera, M. Bauer, & M.T. Hakkinen (2012).
Proceedings of the Invitational Research Symposium on Technology Enhanced Assessments
Center for K–12 Assessment & Performance Management at ETS
This paper presents an overview of the development of a technology enhanced mathematics prototype assessment for the CBAL initiative. Download this paper.
Statistical Report of Fall 2009 CBAL™ Reading Tests
J. Fu, M. Wise, and S. Chung (2012)
ETS Research Memorandum No. RM 12-12
This report presents the statistical results from the administration of two CBAL reading assessments to grade 7 students in nine states in fall 2009. View the full abstract or download this report.
Rethinking K–12 Writing Assessment
P. Deane (2012)
Chapter in N. Elliot & L. Perelman (Editors), Writing Assessment in the 21st Century, Hampton Press.
This book chapter provides a description of the CBAL competency model, outlining its basis in the research literature on writing. Learn more about the book on the publisher's website.
Designing and Evaluating an Interactive Score Report for Students
M. Vezzu, W. VanWinkle, and D. Zapata-Rivera (2012)
ETS Research Memorandum No. RM-12-01
This paper describes an interactive student score report developed for the Cognitively Based Assessment of, for, and as Learning (CBAL) initiative and reports on results from an initial usability study. View the full abstract or download this report.
Designing and Evaluating Score Reports for Particular Audiences
D. Zapata-Rivera (2011)
ETS Research Report No. RR-11-45
This paper presents a framework for creating and assessing score reports for three audiences: teachers, administrators and students as part of a volume of three papers based on workshop presentations at ETS's Score Reporting Conference held on November 4, 2010. View full abstract or download this report.
Constructed-Response Mathematics Tasks Study
J.H. Fife, E.A. Graf, and S. Ohls (2011)
ETS Research Report No. RR-11-35
This report describes potential ways in which selected the CBAL constructed-response mathematics tasks might be revised to reduce construct-irrelevant variance. View full abstract or download this report.
Automated Scoring of CBAL Mathematics Tasks with m-Rater
J.H. Fife (2011)
ETS Research Memorandum No. RM-11-12
This paper presents the automated scoring work done in CBAL Mathematics in 2009 using m-rater, a technology developed at ETS for automated scoring of mathematics items. View the full abstract or download the report.
CBAL: Results From Piloting Innovative K–12 Assessments
R.E. Bennett (2011)
ETS Research Report No. RR-11-23
This report summarizes empirical results from almost 10,000 online administrations of the CBAL summative assessments conducted from 2007 to 2010. View the full abstract or download this report.
The CBAL Reading Assessment: An Approach for Balancing Measurement and Learning Goals
K. M. Sheehan and T. O’Reilly (2011)
ETS Research Report No. RR-11-21
This paper presents a framework for developing new types of reading comprehension assessments that provide evidence about what students know and can do and that help to move learning forward. View the full abstract or download this report.
Automated Scoring Within a Developmental, Cognitive Model of Writing Proficiency
P. Deane, T. Quinlan, and I. Kostin (2011)
ETS Research Report No. RR-11-16
This paper focuses on the potential for using automated scoring techniques to support learning effectively within the CBAL assessments. View the full abstract or download this report
Four Years of Cognitively Based Assessment of, for, and as Learning (CBAL): Learning About Through-Course Assessment (TCA)
J. P. Sabatini, R. E. Bennett, and P. Deane (2011)
Proceedings of the Invitational Research Symposium on Through-Course Summative Assessments
Center for K–12 Assessment & Performance Management at ETS
This paper describes the lessons learned about through-course summative assessment and the reasoning behind some of the design decisions that underlie such assessment in the CBAL. Download this report.
Writing Assessment and Cognition
P. Deane (2011)
ETS Research Report No. RR-11-14
This paper reviews a model that places a strong emphasis on writing as an integrated, socially situated skill. View the full abstract or download this report.
The CBAL Summative Writing Assessment: A Draft Eighth-Grade Design
P. Deane, M. Fowles, D. Baldwin, & H. Persky (2011)
ETS Research Memorandum No. RM-11-01
This paper describes the process and results of developing draft summative writing assessments within the CBAL Initiative. It outlines and reviews four designs, and it briefly discusses initial results from preliminary pilots. View the full abstract or download this report.
Formative Assessment: A Critical Review
R. E. Bennett (2011)
Assessment in Education: Principles, Policy and Practice, Vol. 18, No. 1, pp. 5–25
This paper covers six interrelated issues in formative assessment, several of which motivated the approach taken in the CBAL. View the full abstract or order this report from the publisher.
Cognitively Based Assessment of, for, and as Learning (CBAL): A Preliminary Theory of Action for Summative and Formative Assessment
R.E. Bennett (2010)
Measurement: Interdisciplinary Research & Perspectives, Vol. 8, No. 2–3, pp. 70–91
This paper describes the notion of a theory of action, offers a preliminary version of such a theory for the CBAL initiative, and outlines research necessary to evaluate that theory. View the full abstract or download this report from the publisher.
An Evidence-Centered Approach to Using Assessment Data for Policymakers
J. S. Underwood, D. Zapata-Rivera, & W. VanWinkle (2010)
ETS Research Report No. RR-10-03
District-level policymakers receive reports of student achievement data that are complex, difficult to read and even harder to interpret. In this report, the authors propose an evidence-centered reporting framework in order to design reports that will help policymakers make sense of data. View full abstract or download this report.
Highlights from the Cognitively Based Assessment of, for, and as Learning (CBAL) Project in Mathematics
E. A. Graf, K. Harris, E. Marquez, J. H. Fife, and M. Redman (2010)
ETS Research Spotlight, No. 3, pp. 19–30
This article describes the early design and development stages of the mathematics strand of the Cognitively Based Assessment of, for, and as Learning (CBAL) project. Download the issue of ETS Research Spotlight that contains this article.
c-rater: Automatic Content Scoring for Short Constructed Responses
J. Z. Sukkarieh & J. Blackmore (2009)
Paper in the proceedings of the 22nd Florida Artificial Intelligence Research Society (FLAIRS) Conference
The c-raterSM automated scoring engine, developed at ETS, is a technology for automatic content scoring for short, free-text responses. This paper describes recent developments in this technology. View the full report.
Defining Mathematics Competency in the Service of Cognitively Based Assessment for Grades 6 Through 8
E. A. Graf (2009)
ETS Research Report No. RR-09-42
This report makes recommendations for the development of middle-school mathematics assessment. It discusses how to model mathematical competency at the middle school level, the kinds of evidence that reflect student competency and support future learning, and how to design tasks that elicit evidence. View full abstract or download this report.
Cognitively Based Assessment of, for, and as Learning: A Framework for Assessing Reading Competency
T. O'Reilly & K. M. Sheehan (2009)
ETS Research Report No. RR-09-26
This paper presents the rationale and research base for a reading competency model designed to guide the development of cognitively based assessment of reading comprehension. View full abstract or download this report.
Cognitively Based Assessment of, for, and as Learning: A 21st–Century Approach for Assessing Reading Competency
T. O'Reilly & K. M. Sheehan (2009)
ETS Research Memorandum No. RM-09-04
This paper describes the CBAL system's approach for assessing reading comprehension in an accountability setting. The approach uses evidence-centered design to develop a competency model that drives the development of summative, formative, and professional support aspects of the assessment. To request a copy of this report, send an email to RDWeb@ets.org. Include the title and report number in your request.
Horizontal and Vertical Linking in a Longitudinal Design
F. Rijmen (2009)
ETS Research Memorandum No. RM-09-03
This paper describes two longitudinal data collection designs that may result in substantial reduction of costs. View full abstract or order this report.
Transforming K–12 Assessment: Integrating Accountability Testing, Formative Assessment and Professional Support
R. E. Bennett & D. H. Gitomer (2008)
ETS Research Memorandum No. RM-08-13
This report presents a brief overview of the status of K–12 accountability testing in the United States and describes the CBAL assessment-system model, which is designed to overcome the problems associated with current approaches to accountability testing. Download the report.
Cognitive Models of Writing: Writing Proficiency as a Complex Integrated Skill
P. Deane, N. Odendahl, T. Quinlan, M. Fowles, C. Welsh, & J. Bivens-Tatum (2008)
ETS Research Report No. RR-08-55
This paper undertakes a review of the literature on writing cognition, writing instruction, and writing assessment with the goal of developing a framework and competency model for a new approach to writing assessment. View full abstract or download this report.
Learn about the CBAL Language Arts learning progressions and their relationship to the Common Core State Standards (Flash, 46:20).
Get an overview of the CBAL research program, including how the CBAL assessment prototypes are being used in the classroom (Flash, 8:02).