View in EDS PDF Full Text

Examining Rater Effects in Assessing Learners' Speech and Writing: Insights from ICNALE

Saved in:
Bibliographic Details
Title: Examining Rater Effects in Assessing Learners' Speech and Writing: Insights from ICNALE
Language: English
Authors: Hyunwoo Kim
Source: English Teaching. 2026 81(1):59-80.
Availability: Korea Association of Teachers of English. 6105 English Education Department, Chinju National University of Education, 369beon-gil 3, Jinyangho-ro, Jinju, Gyeongsangnam-do, 52673, Republic of Korea. Tel: +82-42-629-7381; Fax: +82-42-629-7320; e-mail: katejournal29@gmail.com; Web site: https://journal.kate.or.kr/
Peer Reviewed: Y
Page Count: 22
Publication Date: 2026
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Writing Evaluation, Speech Evaluation, Interrater Reliability, Bias, Scoring, English Teachers, English (Second Language), Experience, Evaluators, College Students, Essays, Foreign Countries
Geographic Terms: Asia
ISSN: 1017-7108
2671-9312
Abstract: This study investigated rater effects in the ICNALE Global Rating Archives (GRA), which contain ratings from 160 raters on 140 speeches and 140 essays across 10 rating criteria. After cleaning the raw dataset, two separate three-facet partial credit models were fitted to the speech and essay datasets to examine rater consistency, and a bias analysis was conducted to investigate the relationship between raters' prior rating experience and severity. The results indicated that six raters in the speech ratings exhibited overly inconsistent rating patterns, but no rater in the essay ratings did. It was also found that raters' prior rating experience was significantly associated with the complexity, involvement, and accuracy criteria in the speech ratings and with the complexity criterion in the essay ratings. However, the corresponding effect sizes for these interactions were trivial, indicating limited practical impact. The educational implication is that rigorous rater training could enhance rater consistency among both pre-service and less-experienced in-service English teachers in speaking and writing assessment.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1504902
Database: ERIC
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwECMUOM3aCAwV1VC8vvKuUFAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDEYaE37JaKtsul0HHQIBEICBm3D-WVv8t-8wXlqlgaD-7vRzrXXhNRkEXkpGdbp7XuzOLBDVdykbPtdj2TZ8RKXbORzdz326e77wOPOk75mPLrXkNN-nwYxvrrtCmmbCCVANDrMtdBiUs5PSYcRfIeVRFpT_Y75La42OdrBc5m_f9IQzpNOkUHsIkT09BhtFTi-iOkfKOxY8Z_fzXiDOderfBpb3BRCoMq2Ptl-X
Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1504902
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1504902
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Examining Rater Effects in Assessing Learners' Speech and Writing: Insights from ICNALE
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Hyunwoo+Kim%22">Hyunwoo Kim</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22English+Teaching%22"><i>English Teaching</i></searchLink>. 2026 81(1):59-80.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Korea Association of Teachers of English. 6105 English Education Department, Chinju National University of Education, 369beon-gil 3, Jinyangho-ro, Jinju, Gyeongsangnam-do, 52673, Republic of Korea. Tel: +82-42-629-7381; Fax: +82-42-629-7320; e-mail: katejournal29@gmail.com; Web site: https://journal.kate.or.kr/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 22
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Speech+Evaluation%22">Speech Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Bias%22">Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22English+Teachers%22">English Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22English+%28Second+Language%29%22">English (Second Language)</searchLink><br /><searchLink fieldCode="DE" term="%22Experience%22">Experience</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Essays%22">Essays</searchLink><br /><searchLink fieldCode="DE" term="%22Foreign+Countries%22">Foreign Countries</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Asia%22">Asia</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1017-7108<br />2671-9312
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study investigated rater effects in the ICNALE Global Rating Archives (GRA), which contain ratings from 160 raters on 140 speeches and 140 essays across 10 rating criteria. After cleaning the raw dataset, two separate three-facet partial credit models were fitted to the speech and essay datasets to examine rater consistency, and a bias analysis was conducted to investigate the relationship between raters' prior rating experience and severity. The results indicated that six raters in the speech ratings exhibited overly inconsistent rating patterns, but no rater in the essay ratings did. It was also found that raters' prior rating experience was significantly associated with the complexity, involvement, and accuracy criteria in the speech ratings and with the complexity criterion in the essay ratings. However, the corresponding effect sizes for these interactions were trivial, indicating limited practical impact. The educational implication is that rigorous rater training could enhance rater consistency among both pre-service and less-experienced in-service English teachers in speaking and writing assessment.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1504902
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1504902
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
        StartPage: 59
    Subjects:
      – SubjectFull: Writing Evaluation
        Type: general
      – SubjectFull: Speech Evaluation
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: Bias
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: English Teachers
        Type: general
      – SubjectFull: English (Second Language)
        Type: general
      – SubjectFull: Experience
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: College Students
        Type: general
      – SubjectFull: Essays
        Type: general
      – SubjectFull: Foreign Countries
        Type: general
      – SubjectFull: Asia
        Type: general
    Titles:
      – TitleFull: Examining Rater Effects in Assessing Learners' Speech and Writing: Insights from ICNALE
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Hyunwoo Kim
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 1017-7108
            – Type: issn-electronic
              Value: 2671-9312
          Numbering:
            – Type: volume
              Value: 81
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: English Teaching
              Type: main
ResultId 1