Effects of Item Disclosure on TOEFL® Performance

Author(s):
Hale, Gordon A.; Angelis, Paul J.; Thibodeau, Lawrence A.
Publication Year:
1980
Report Number:
RR-80-34, TOEFL-RR-08
Source:
Document Type:
Subject/Key Words:
Coaching English tests foreign students item analysis performance factors

Abstract

The Test of English as a Foreign Language™ (TOEFL®) forms previously used in international or special centers administrations have been given to institutions for use in their instructional testing programs. Now, however, new legislation requires that standardized tests be disclosed after administration. Alternate procedures need to instituted. One alternative is to combine items from previously used forms, possibly including some that have been disclosed, into new forms for use by institutions. Before doing this, it is necessary to know how TOEFL performance would be influenced if candidates could see some of the test items before the test was given. The present study is experimental. A number of disclosed forms were made available to foreign students in intensive English language programs. Later, the pupils were given a special TOEFL consisting of items from these forms and a TOEFL consisting of undisclosed items. Effects of item disclosure were determined by contrasting performance on these two tests. The students were given copies of the disclosed forms, and their language instructors discussed a portion of the items on these forms in class. Separate analysis of the test performance for items that had been disclosed but not discussed in class showed the extent to which students benefit from disclosed items when left on their own initiative to study them. An additional variable was the size of the disclosed item pool. It was hypothesized that if pupils need to cover a large amount of test forms, they will be less likely to benefit from disclosure than if they need only cover a small number of test forms. To test this, items to appear on the test were spread through six disclosed forms for students in eight institutions and through 12 forms for students in eight other institutions. A significant disclosure effect was observed. Scores for the test containing disclosed items were higher than those for the test containing undisclosed items. This shows that pupils studied the disclosed forms and remembered specific items. The effect of disclosure was greater for students who received six disclosed forms than for those who received 12 forms. Student questionnaire information suggested that the time spent studying the disclosed material was unaffected by the number of forms. Additional evaluations showed disclosure effects for items in forms that were not discussed in class as well as for items that were discussed. The effect of disclosure was not due only to the class experience, as the students apparently studied the disclosed forms on their own. A disclosure effect was seen for each of three main language groups and for every item type.

Read More