angle-up angle-right angle-down angle-left close user menu open menu closed search globe bars phone store

Frequently Asked Questions About the Criterion® Online Writing Evaluation Service

Teacher and Student Use

What is a Criterion® service score?
A Criterion score is a holistic score based on the traits of word choice, convention and fluency/organization usually on a 4- or 6-point scale that is given to an essay.

 

  • The Criterion service scoring compares a student's writing to essays written and evaluated by writing instructors.
  • The essays used to build the scoring models have been scored by trained readers and were written by students under timed-testing conditions.
  • The Criterion service score takes content relevance into account by analyzing the degree of similarity between prompt-specific vocabulary and that of the response to prompts supplied by ETS.
In what ways can the Criterion scores be used?
  • Placement. Students' scores on a Criterion service-scored essay may be used as one indicator in assigning them to classes, in combination with other indicators. The electronic score should not be the sole basis for a placement decision. It is best to combine a Criterion service score with the score of a human reader in the same way that institutions combine scores from two different human readers. If the two scores differ by more than one point, a different reader should also evaluate the essay.
  • Exit Assessment. Institutions can use the Criterion service scores for exit testing — combining the institution's assessment tool and the Criterion service score with the score from a reader in the same way they combine scores from two different readers.
  • Benchmark Testing. Instructors can assign the Criterion service-scored essays at specified points during an academic term while aligning common core standards. Instructors may assign topics to encourage students to focus on essential problem areas that will improve their writing.
  • Writing Labs. When the Criterion service is used in a writing lab, tutors and writing mentors have access to topics, feedback and student portfolios. The interactive features of the Criterion service promote the communication of instructors about student progress between classroom learning and writing lab support. Use of the Criterion service in a writing lab facilitates writing across the curriculum when students use the lab to check in-progress writing for all of their classes.
  • Writing Improvement and Remediation. Instructors assign the Criterion service standard topics or use their own topics to give students opportunities for additional writing practice and developing writing skills in students. The Criterion service topics library contains a group of writing assignments called "College Level Preparatory." These topics are graded against a lower-level scoring rubric and can be assigned to gradually move incoming freshmen up to the first-year writing level.
What are Criterion service topics?

The Criterion® Online Writing Evaluation Service Topics Library for K–12 Education includes grade level appropriate vocabulary, writing prompts for grades 4–12 that appeal to student interests. Criterion essay topics are constructed to elicit writing in various modes that include persuasive, expository, descriptive and narrative. Each topic may be scored on either a 6-point or 4-point scale and the associated rubrics are shown with each prompt.

Currently, there are over 200 K–12 topics and over 100 College Level topics appropriate for first- and second-year writing courses, practice and placement. There are over 50 additional prompts available in the library including College Preparatory, GRE®, Praxis®, TOEFL® and TOEFL Junior® test topics. Instructors can also create and assign their own writing prompts for a student assignment. Because instructors can create their own topics, the potential for a topic library is endless.

How is the Criterion service helpful to students?

Students find out immediately how their work compares to a standard and what they should do to improve it. The Criterion service also provides an environment for writing and revision that students can use independently, 24 hours a day. This environment, coupled with the opportunity for instant feedback, provides the directed writing practice that is so beneficial for students, especially in improving English-writing skills.

As students become more sophisticated writers, they start to think about the organizational structure in their writing. Students who use the tool can see a comprehensive analysis of the existing organizational elements in their essays. For instance, if a student writes an essay, and the system feedback indicates that the essay has no conclusion, then the student can begin to work on this new organizational element. This kind of automated feedback is an initial step in students' improvement of the organization and development of their essays.

For additional quick tips, students can review the Student Quick Access Guide for more information.

Can educators implement limits?

Educators have easy and secure access to the online teaching material and each student's portfolio of essays, diagnostic reports and scores, as well as summary information on the performance of entire classes.

  • Instructors can elect to report all, some or none of the feedback analysis.
  • When creating an assignment, instructors turn the score analysis feature on or off, as well as select which diagnostic feedback to report.
  • Instructors can limit access when selecting assignment options. For example, the date and time an assignment is available are selected by instructors during setup. They can also limit how many times a student can write and revise an assignment.
  • Educators can block students from seeing their scores — and may choose to do so if they use the Criterion service for benchmarking.
Does the Criterion service handle an unusual writing style or stifle creative writing among students?

The Criterion service looks for specific features of syntax, organization and vocabulary. If the essay under consideration is not sufficiently similar to those in its database of already-scored essays, the Criterion service posts a warning, called an Advisory, saying that it is unable to give an accurate score. Advisories usually result from essays that are too brief or those in which the vocabulary is unusual, or the content is off topic.

The Criterion service is designed to be used for evaluating writing done under testing conditions — situations in which even the most creative writers concentrate on straightforward and competent writing.

What is the Writer's Handbook?

The Writer's Handbook is an intuitive online tool that a student can access while reviewing diagnostic feedback. It explains every error or feature reported by defining it and providing examples of correct and incorrect use. There are five Writer's Handbook versions available:

  • Elementary
  • Middle School
  • Descriptive
  • High School/College
  • ELL

There are also four bilingual versions available:

  • Spanish/English
  • Simplified Chinese/English
  • Japanese/English
  • Korean/English
How is the Criterion service feedback different from the Microsoft Word® Spelling and Grammar tool?

The Microsoft Word Spelling and Grammar tool can provide writers with a quick analysis of common errors. The Criterion service, as an instructional tool used to improve writing, targets more precise feedback. Research shows that the spelling error detection and correction module in the Criterion service has better precision than the spelling error detection and correction module used in Microsoft Word. We continually strive through research and user input to improve the precision of all our feedback categories.

Scoring and Analysis

How does the Criterion service come up with its scores?

The Criterion service is based on a technology called the e-rater® Automated Scoring Engine that was developed by ETS. The e-rater scoring engine compares the new essay to samples of essays previously scored by readers, looking for similarities in sentence structure, organization and vocabulary.

Can the Criterion service score essays on other topics?

Using the Scored Instructor Topic feature, teachers can create their own topics that are parallel to the Criterion service library prompts, and the students' essays will receive Criterion scores upon completion. A link in the Criterion service provides step-by-step instructions on how to create either a persuasive or expository topic that can be scored.

Does the computer's score agree with the score of an instructor or reader?

ETS researchers generally found either exact or adjacent agreement (within one point) between the Criterion service scores and those of a trained essay reader or instructor. Both used the same scoring guidelines and scoring system.

What information is reported to students and teachers?

Students get diagnostic feedback, as well as a holistic evaluation, each time they submit an essay. Educators have easy and secure access to each student's portfolio of essays, diagnostic reports and scores, as well as summary information on the performance of entire classes.

Can the Criterion service catch cheating or plagiarism?

The Criterion service simply evaluates the essay. The e-rater scoring engine cannot really understand English, and it can be fooled by an illogical, but well-written, argument. Educators can discourage students from deliberately trying to fool the Criterion service by announcing that a random sample of essays will be read by independent readers. The Criterion service will also display an Advisory along with the e-rater score when an essay displays certain characteristics. Advisories usually result from essays that are too brief or those in which the vocabulary is unusual or the content is off-topic.

It is up to the institution to ensure that students are working independently and submitting their own work. Instructors can opt to display a writer's sample for some topics on the "Create Assignment" screen. Students can then view the samples and refer to them while they write their own essays. The sample essays are in a read-only format and cannot be copied and pasted into another document.

Why do educators value the Criterion service automated analysis of essay-based organizational elements in student essays?

There is now broad acceptance of automated essay scoring technology for large-scale assessment and classroom instruction. This technology is a helpful supplement to traditional teacher instruction. Specifically, it is more effective for students to receive feedback that refers explicitly to their own writing rather than just general feedback. Instructors and educational researchers encourage the development of improved essay evaluation applications that not only generate a numerical rating for an essay, but also analyze grammar, usage, mechanics and discourse structure. The Criterion service capability to analyze organizational elements serves as a critical complement to other tools in the application that provide feedback related to grammar, usage, mechanics and style features in student essays.

How did the system learn how to do the analysis?

Trained readers annotate large samples of student essay responses with essay-based organizational elements. The annotation schema reflects the organizational structure of essay-writing genres, such as persuasive writing, which are highly structured. The increased use of automated essay-scoring technology allows for the collection of a large corpus of students' essay responses that we use for annotation purposes.

Which organizational elements are analyzed?

This cutting edge, first-of-its-kind technology employs machine learning to identify organizational elements in student essays, including:

  • introductory or background material
  • thesis statements
  • main ideas
  • supporting ideas
  • conclusions

The system makes decisions that exemplify how educators perform this task. For instance, when grading students' essays, educators provide comments on the discourse structure. Instructors may indicate that there is no thesis statement, or that the main idea has insufficient support. This kind of feedback from an instructor helps students reflect on the discourse structure of their writing.

Understanding Organization and Development Feedback

How does the automated system make decisions about text segments in a student essay and the corresponding organizational labels?

The algorithm developed to automatically identify essay-based organizational elements is based on samples of teacher-annotated essay data. Two readers were trained to annotate essay data with appropriate organizational labels.

How does the system label each individual sentence?

Sometimes multiple sentences are associated with a single organizational element, and the entire block of text is highlighted and appears to be assigned a single label. In fact, each sentence is labeled individually. Many features, including word usage, rhetorical strategy information, possible sequence of organizational elements and syntactic information are used to determine the final organizational label.

Technology Overview and Understanding

What is the technology used in the e-rater engine scoring?

The e-rater scoring engine is an application of Natural Language Processing (NLP), a field of computer technology that uses computational methods to analyze characteristics of text. Researchers have been using NLP for more than 50 years to translate text from one language to another and to summarize text. Internet search engines currently use NLP to retrieve information.

The e-rater scoring engine uses NLP to identify the features of the human-scored essays in its sample collection and store them — with their associated weights — in a database. When the e-rater engine evaluates a new essay, it compares its features to those in the database in order to assign a score. Because the e-rater scoring engine is not doing any actual reading, the validity of its scoring depends on the scoring of the sample essays from which the e-rater database is created.

The research papers and additional resources on the ETS website are sources of more information about the Criterion service and its underlying technology.

Must students be connected to the internet to use the Criterion service?

Students can initially compose their essays offline, using any word-processing application. However, they will ultimately need an internet connection to be able to cut and paste their essays into the Criterion service essay submission box so their work can be scored and analyzed. For assignments that are timed, essays should be composed online only to ensure accountability by all students and to accurately reflect their writing skills in this environment.

What technical requirements must a user have to access the Criterion service site?

The Criterion service is available 24 hours a day and only requires an internet connection, a web browser and is PC- and Mac®-compatible. It can also be used on the iPad®, but an external keyboard is recommended. For a complete description of minimum and recommended standards and network configuration suggestions, please refer to the System Requirements Sheet.

Can I import student identifiers from my data management system and save my data?

The Criterion service has export and import capabilities for administrators at several levels. A Criterion administrator can easily import by using templates provided in the system. The export features easily allow users to create export files, and an archive portfolio feature used to create export files in a comma-delimited format (.csv) that can be opened by most text editors and spreadsheet programs.

Details instructions are provided in both the HELP text and the Criterion® User Manual and Administrator Supplement.