View in EDS

Setting Standards for a Diagnostic Test of Aviation English for Student Pilots

Saved in:
Bibliographic Details
Title: Setting Standards for a Diagnostic Test of Aviation English for Student Pilots
Language: English
Authors: Maria Treadaway (ORCID 0000-0001-5610-5790), John Read (ORCID 0000-0002-2774-4398)
Source: Language Testing. 2024 41(3):557-583.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 27
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Descriptors: Standard Setting, Diagnostic Tests, High Stakes Tests, English for Special Purposes, Aviation Education, Flight Training, Language Tests, Language Proficiency, Rating Scales, Test Construction, Test Validity, Cutting Scores, Reading Tests, Listening Comprehension Tests, Test Reliability
DOI: 10.1177/02655322231224051
ISSN: 0265-5322
1477-0946
Abstract: Standard-setting is an essential component of test development, supporting the meaningfulness and appropriate interpretation of test scores. However, in the high-stakes testing environment of aviation, standard-setting studies are underexplored. To address this gap, we document two stages in the standard-setting procedures for the Overseas Flight Training Preparation Test (OFTPT), a diagnostic English test for ab initio pilots, aligned to the International Civil Aviation Organization (ICAO)'s Language Proficiency Rating Scale (LPRS). Performance-level descriptors (PLDs) were empirically generated in Stage 1 in collaboration with six subject matter experts (SMEs). These PLDs made explicit the correspondence between linguistic performance levels within the target language use domain and the ICAO scale. Findings suggest that the ICAO scale is not fine-grained enough to distinguish levels of linguistic readiness among ab initio pilots, nor does it adequately reflect the knowledge, skills, and abilities valued by SMEs within this domain. In Stage 2, 12 SMEs were recruited to set standards and were divided into two groups to investigate the replicability of Ebel method standard-setting procedures. Cut scores were determined for the OFTPT reading and listening tests, which were inferentially linked to the LPRS. There were no significant differences in the cut scores arrived at by both groups and reliability was excellent, suggesting that test users can have confidence in the standards set.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1436409
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1436409
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Setting Standards for a Diagnostic Test of Aviation English for Student Pilots
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Maria+Treadaway%22">Maria Treadaway</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-5610-5790">0000-0001-5610-5790</externalLink>)<br /><searchLink fieldCode="AR" term="%22John+Read%22">John Read</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-2774-4398">0000-0002-2774-4398</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Language+Testing%22"><i>Language Testing</i></searchLink>. 2024 41(3):557-583.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 27
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Standard+Setting%22">Standard Setting</searchLink><br /><searchLink fieldCode="DE" term="%22Diagnostic+Tests%22">Diagnostic Tests</searchLink><br /><searchLink fieldCode="DE" term="%22High+Stakes+Tests%22">High Stakes Tests</searchLink><br /><searchLink fieldCode="DE" term="%22English+for+Special+Purposes%22">English for Special Purposes</searchLink><br /><searchLink fieldCode="DE" term="%22Aviation+Education%22">Aviation Education</searchLink><br /><searchLink fieldCode="DE" term="%22Flight+Training%22">Flight Training</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Tests%22">Language Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Proficiency%22">Language Proficiency</searchLink><br /><searchLink fieldCode="DE" term="%22Rating+Scales%22">Rating Scales</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Construction%22">Test Construction</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Validity%22">Test Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Cutting+Scores%22">Cutting Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Reading+Tests%22">Reading Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Listening+Comprehension+Tests%22">Listening Comprehension Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reliability%22">Test Reliability</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/02655322231224051
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0265-5322<br />1477-0946
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Standard-setting is an essential component of test development, supporting the meaningfulness and appropriate interpretation of test scores. However, in the high-stakes testing environment of aviation, standard-setting studies are underexplored. To address this gap, we document two stages in the standard-setting procedures for the Overseas Flight Training Preparation Test (OFTPT), a diagnostic English test for ab initio pilots, aligned to the International Civil Aviation Organization (ICAO)'s Language Proficiency Rating Scale (LPRS). Performance-level descriptors (PLDs) were empirically generated in Stage 1 in collaboration with six subject matter experts (SMEs). These PLDs made explicit the correspondence between linguistic performance levels within the target language use domain and the ICAO scale. Findings suggest that the ICAO scale is not fine-grained enough to distinguish levels of linguistic readiness among ab initio pilots, nor does it adequately reflect the knowledge, skills, and abilities valued by SMEs within this domain. In Stage 2, 12 SMEs were recruited to set standards and were divided into two groups to investigate the replicability of Ebel method standard-setting procedures. Cut scores were determined for the OFTPT reading and listening tests, which were inferentially linked to the LPRS. There were no significant differences in the cut scores arrived at by both groups and reliability was excellent, suggesting that test users can have confidence in the standards set.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1436409
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1436409
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/02655322231224051
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 27
        StartPage: 557
    Subjects:
      – SubjectFull: Standard Setting
        Type: general
      – SubjectFull: Diagnostic Tests
        Type: general
      – SubjectFull: High Stakes Tests
        Type: general
      – SubjectFull: English for Special Purposes
        Type: general
      – SubjectFull: Aviation Education
        Type: general
      – SubjectFull: Flight Training
        Type: general
      – SubjectFull: Language Tests
        Type: general
      – SubjectFull: Language Proficiency
        Type: general
      – SubjectFull: Rating Scales
        Type: general
      – SubjectFull: Test Construction
        Type: general
      – SubjectFull: Test Validity
        Type: general
      – SubjectFull: Cutting Scores
        Type: general
      – SubjectFull: Reading Tests
        Type: general
      – SubjectFull: Listening Comprehension Tests
        Type: general
      – SubjectFull: Test Reliability
        Type: general
    Titles:
      – TitleFull: Setting Standards for a Diagnostic Test of Aviation English for Student Pilots
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Maria Treadaway
      – PersonEntity:
          Name:
            NameFull: John Read
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 07
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 0265-5322
            – Type: issn-electronic
              Value: 1477-0946
          Numbering:
            – Type: volume
              Value: 41
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Language Testing
              Type: main
ResultId 1