View in EDS

Do Source Use Features Impact Raters' Judgment of Argumentation? An Experimental Study

Saved in:
Bibliographic Details
Title: Do Source Use Features Impact Raters' Judgment of Argumentation? An Experimental Study
Language: English
Authors: Ping-Lin Chuang (ORCID 0000-0003-2907-5324)
Source: Language Testing. 2025 42(1):48-72.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 25
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Interrater Reliability, Evaluators, Information Sources, Primary Sources, Persuasive Discourse, Evaluative Thinking, Evaluation Criteria, Evaluation Methods, Second Language Learning, Writing Tests, Achievement Rating, Causal Models
Geographic Terms: Illinois (Urbana)
DOI: 10.1177/02655322241263629
ISSN: 0265-5322
1477-0946
Abstract: This experimental study explores how source use features impact raters' judgment of argumentation in a second language (L2) integrated writing test. One hundred four experienced and novice raters were recruited to complete a rating task that simulated the scoring assignment of a local English Placement Test (EPT). Sixty written responses were adapted from essays written by EPT test-takers. These responses were crafted to reflect different conditions of source use features, namely source use quantity and quality. Rater scores were analyzed using the many-facet Rasch model and mixed two-way analyses of variance (ANOVAs) to examine how they are affected by source use features and rater experience. Results show that source use features impacted the argumentation scores assigned by raters. Paragraphs with more source text ideas that are better incorporated received the highest argumentation scores, and vice versa for those with limited, poorly integrated source information. Rater experience impacted scores but did not influence rater performance meaningfully. The findings of this study connect specific source use features with raters' evaluation of argumentation, helping to further disentangle the relationships among examinee performance, rater decision, and task features of integrated argumentative writing tests. They also provide meaningful implications for writing assessment research and practices.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1453550
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1453550
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Do Source Use Features Impact Raters' Judgment of Argumentation? An Experimental Study
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Ping-Lin+Chuang%22">Ping-Lin Chuang</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-2907-5324">0000-0003-2907-5324</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Language+Testing%22"><i>Language Testing</i></searchLink>. 2025 42(1):48-72.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 25
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluators%22">Evaluators</searchLink><br /><searchLink fieldCode="DE" term="%22Information+Sources%22">Information Sources</searchLink><br /><searchLink fieldCode="DE" term="%22Primary+Sources%22">Primary Sources</searchLink><br /><searchLink fieldCode="DE" term="%22Persuasive+Discourse%22">Persuasive Discourse</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluative+Thinking%22">Evaluative Thinking</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Criteria%22">Evaluation Criteria</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+Methods%22">Evaluation Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Learning%22">Second Language Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Tests%22">Writing Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Achievement+Rating%22">Achievement Rating</searchLink><br /><searchLink fieldCode="DE" term="%22Causal+Models%22">Causal Models</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Illinois+%28Urbana%29%22">Illinois (Urbana)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/02655322241263629
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0265-5322<br />1477-0946
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This experimental study explores how source use features impact raters' judgment of argumentation in a second language (L2) integrated writing test. One hundred four experienced and novice raters were recruited to complete a rating task that simulated the scoring assignment of a local English Placement Test (EPT). Sixty written responses were adapted from essays written by EPT test-takers. These responses were crafted to reflect different conditions of source use features, namely source use quantity and quality. Rater scores were analyzed using the many-facet Rasch model and mixed two-way analyses of variance (ANOVAs) to examine how they are affected by source use features and rater experience. Results show that source use features impacted the argumentation scores assigned by raters. Paragraphs with more source text ideas that are better incorporated received the highest argumentation scores, and vice versa for those with limited, poorly integrated source information. Rater experience impacted scores but did not influence rater performance meaningfully. The findings of this study connect specific source use features with raters' evaluation of argumentation, helping to further disentangle the relationships among examinee performance, rater decision, and task features of integrated argumentative writing tests. They also provide meaningful implications for writing assessment research and practices.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1453550
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1453550
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/02655322241263629
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 25
        StartPage: 48
    Subjects:
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: Evaluators
        Type: general
      – SubjectFull: Information Sources
        Type: general
      – SubjectFull: Primary Sources
        Type: general
      – SubjectFull: Persuasive Discourse
        Type: general
      – SubjectFull: Evaluative Thinking
        Type: general
      – SubjectFull: Evaluation Criteria
        Type: general
      – SubjectFull: Evaluation Methods
        Type: general
      – SubjectFull: Second Language Learning
        Type: general
      – SubjectFull: Writing Tests
        Type: general
      – SubjectFull: Achievement Rating
        Type: general
      – SubjectFull: Causal Models
        Type: general
      – SubjectFull: Illinois (Urbana)
        Type: general
    Titles:
      – TitleFull: Do Source Use Features Impact Raters' Judgment of Argumentation? An Experimental Study
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Ping-Lin Chuang
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 0265-5322
            – Type: issn-electronic
              Value: 1477-0946
          Numbering:
            – Type: volume
              Value: 42
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Language Testing
              Type: main
ResultId 1