Flying High in Language Learning: How Dr. Maria Treadaway’s Research Could Impact English Language Assessment
March 24, 2023
Dr. Maria Treadaway has been named the recipient of this year’s Jacqueline Ross TOEFL® Dissertation Award for her dissertation, “Developing and Validating a Diagnostic Language Test for Ab Initio Cadet Pilots.” The Jacqueline Ross TOEFL Dissertation Award recognizes doctoral dissertation research that makes a significant and original contribution to knowledge about second or foreign-language tests and testing and/or the use and development of such tests and testing. In this Q&A, Dr. Treadaway will share her research on the significance of findings for the aviation industry and language testing field and how it contributes to the field of English-language learning and assessment, among others.
Can you tell us about your inspiration for conducting research in the vital area of language testing in aviation?
Dr. Maria Treadaway: Ever since receiving Richard Scarry’s Great Big Air Book for my fifth birthday, I was fascinated by aircraft and aviation. I eventually attained my commercial pilot’s license in 2001, but following 9/11, I decided that I would keep flying as a hobby rather than pursuing it professionally. Instead, I took up English-language teaching, going on to complete a master’s in applied linguistics. My minor dissertation combined an interest in aviation and assessment and I investigated the effect of the rate of speech on pilot comprehension in nonroutine situations. During this, I was incredibly lucky to work with Professor Emeritus John Read because his supervision combined both aviation and assessment expertise. I realized in my master’s research that there was a real need for research into aviation language testing and because John agreed to work with me, I knew I would be supported in my Ph.D.. In fact, I am in the very privileged position of being John’s last doctoral student and this award is as much for him as me. So, in my study, I was able to combine my love of aviation, the English language and assessment.
Can you elaborate on the significance of your research findings for the aviation industry and language testing field?
Dr. Maria Treadaway: My study was an applied project to develop a test to measure the linguistic readiness of ab initio cadet pilots from non-English-speaking backgrounds to engage successfully in their practical flight training programmes conducted in English. The test is called the Overseas Flight Training Preparation Test or OFTPT. In case you were wondering, ab initio means from the beginning, so we’re talking about student pilots who have completed their ground school theoretical training and are ready to transition into practical flight training and cadet is referring to the fact that a lot of the participants were being sponsored in their study by an airline.
The motivation for this project emerged from the current context of language testing within aviation training environments. At present, there are no standardized measures of language ability to assess ab initio cadet pilots from non-English-speaking backgrounds who are entering practical flight training in English-speaking countries. Instead, each individual training organization adopts their own testing and entry requirements, which results in an ad hoc approach to assessing linguistic readiness in contrast to the otherwise robust and standardized assessment of skills and abilities in other areas of their training. Consequently, the aviation education sector has signaled the need for a valid, reliable and standardized measurement tool, calibrated to the International Civil Aviation Organization’s (ICAO) language proficiency rating scale (this is the international measure for pilot and air traffic controller language proficiency), to assist in establishing comparable quality standards across organizations with the aim of minimizing the negative consequences experienced by student pilots, sponsoring airlines and training organizations when students are granted entry onto a programme when they are not linguistically ready. So in terms of the significance of my research, it may represent the measurement tool that the industry is seeking because it was motivated by the real-world needs of the industry.
Data from the investigation of the Target Language Use (TLU) domain (Phase 1) and the standard setting session (Phase 4) indicate that the OFTPT is successful in capturing what subject matter experts (SMEs) within this domain (flying instructors, air traffic controllers and aviation English-language teachers) value in terms of test content. The item analysis in test trials (Phase 3) shows that the operationalization of the construct (Phase 2) was successful in that the test discriminates well and was at a suitable level of difficulty. During standard setting with SMEs (Phase 4), there was also a considerable degree of homogeneity regarding the levels of linguistic performance SMEs expected from Non-Native English Speaking (NNES) ab initios, suggesting that confidence can be held in the threshold levels of performance articulated in the performance level descriptors and in the cut scores, which were inferentially linked to the ICAO language proficiency rating scale. In sum, each phase contributed to identifying a standard determined by SMEs in a test developed specifically for this TLU domain. This is the first research-based benchmark for entry into flight training, measured against the ICAO scale. It’s a start and I know that many other people are working towards improving standards in this field and I hope that others can and will build on my work.
How do you envision your research contributing to the field of English-language learning and assessment?
Dr. Maria Treadaway: In terms of a practical contribution, the main output of the project is a fully operational test, administered online, which specifically targets the needs of ab initio pilots embarking on their practical flight training and aligned with the ICAO Language Proficiency Rating scale. Test scores can be interpreted as levels of readiness to engage in practical flight training (so a student is either ready, minimally ready or not ready) and these levels of readiness have been empirically linked to the ICAO rating scale, which as I said previously is the first time that threshold levels of language proficiency within this TLU domain have been linked to the scale.
The test also focusses on the important diagnostic function of testing in this environment. In the score report, test takers receive detailed feedback on their test performance that could help them to identify their strengths and weaknesses in order to improve their aviation language skills. On the companion test website, I curated learning resources that test takers can freely access to improve their weaknesses and build on their strengths. So there is potential for a positive washback on language learning outcomes, where learners are developing the knowledge, skills and abilities they need in their real-world domains and being tested on the same abilities in the OFTPT. This alignment between learning and assessment was very important to me when designing my aviation language test. The language test tasks are very closely aligned with tasks in the target language use domain of the students, so that learning and assessment flow into each other. I think this is so important in motivating learners and increasing their engagement with the language they need to function within their target language use domain, in this case, practical flight training.
In terms of the theoretical contribution, this study is the first work to merge two well-established approaches to validation in assessment: that Bachman and Palmer’s Assessment Use argument, and O’Sullivan and Weir’s socio-cognitive framework 2011. As such, the extended descriptions of the design and operationalization of the OFTPT in Phase 2 (in Chapter 5 of my dissertation) could serve as a useful blueprint for ESP/LSP test developers and particularly aviation test developers, because the chapter also includes detailed test specifications and a rationale for each module of the OFTPT (Listening, Reading and Speaking).
How has receiving the Jacqueline Ross TOEFL® Dissertation Award impacted your career and research trajectory?
Dr. Maria Treadaway: First and foremost, receiving this award has brought a significant level of recognition within my school and faculty at the University and Auckland and within the language testing community in Australia and New Zealand and now with this interview perhaps further afield as well! I am hoping that this increases the visibility for my doctoral project, perhaps resulting in some real-world applications of the OFTPT within aviation training organizations. Also, because this is an international award, it adds to the credibility of my research, which may help with uptake of the test within aviation training organizations and with securing future funding for further validation and research. In addition to recognition, the award comes with financial support that I will use to disseminate my research by attending conferences and other professional development opportunities. This recognition is already translating into increased opportunities to collaborate with other researchers from Australia, Brazil and the United States which in turn, I hope will lead to new research avenues and further advancement of the field. So, I think this award has helped me as a platform to share my work with a broader audience.
Can you speak about the importance of recognizing and supporting research in language testing?
Recognizing and supporting research in language testing is hugely important for several reasons. First, research in language testing helps to ensure that language tests are fair, reliable and valid, which is particularly important in contexts where language proficiency is used for high-stakes purposes, such as aviation or immigration. If we can improve the accuracy of language assessment, we can have a positive impact on individuals' educational and professional opportunities, as well as on society as a whole. Without research, we may not be able to capture the target construct or accurately measure language proficiency, leading to inaccurate evaluations of language learners and unintended negative consequences for all stakeholders. Advancements in technology can support research in language testing, allowing for the development of innovative assessment methods and tools that can better capture language proficiency and that can improve the testing experience for test takers, perhaps by making tests more accessible and equitable. Finally, research in language testing can help inform language policy and decision making, leading to more effective language education programs and improved outcomes for language learners. So, research is pretty important!
What were some of the biggest challenges you faced during your doctoral project, and how did you overcome them?
Dr. Maria Treadaway: A lot of my Ph.D. was done during the COVID-19 pandemic, so there were unique challenges that made the process more complex, but also presented me with new opportunities to explore innovative solutions using digital and collaborative tools. Specific challenges involved project managing all of the small tasks that needed to be completed, developing and maintaining connections within the aviation training industry, recruiting participants (especially with the restrictions of governments and social distancing) and the general need for virtual interviews and online data collection. Occasionally, I was unable to access physical resources from the library, which made time management and coordinating the flow of the research activities difficult at times. Apart from COVID, probably like most Ph.D. students, I found maintaining motivation, especially during the writing process most challenging. The way I overcame this was to build in recreation to my schedule — walks, visits with family and friends and for work. This balance was necessary to sustain the extended focus I needed. I also kept a log of the activities I was working on, the intentions/outcomes that I was working towards and a reflection on how I was doing and the time I spent working on the activities. This log of time and activity was essential in helping me to remain focused, motivated and on-task while working remotely and in isolation.
Lastly, what message would you like to share with aspiring researchers in this field?
Dr. Maria Treadaway: The implications of language testing for the aviation industry are far-reaching, and I think there is a lot of scope for further research to ensure that language testing in this field is effective, up-to-date and that tests are developed according to professional standards, with validation evidence to support the inferences of the test scores. So for those that are doing research and perhaps are feeling the challenges of this, remember that language testing is of paramount importance for the aviation industry in ensuring safety, improving communication and in turn enhancing situational awareness, and improved decision making within the cockpit. Keep going and know that what you’re working on counts!