View in EDS

Using Automated Analysis to Assess Middle School Students' Competence with Scientific Argumentation

Saved in:
Bibliographic Details
Title: Using Automated Analysis to Assess Middle School Students' Competence with Scientific Argumentation
Language: English
Authors: Christopher D. Wilson (ORCID 0000-0002-7342-992X), Kevin C. Haudek (ORCID 0000-0003-1422-6038), Jonathan F. Osborne (ORCID 0000-0001-8096-208X), Zoë E. Buck Bracey (ORCID 0000-0003-4826-3582), Tina Cheuk (ORCID 0000-0001-7841-2492), Brian M. Donovan (ORCID 0000-0003-2329-4198), Molly A. M. Stuhlsatz, Marisol M. Santiago, Xiaoming Zhai (ORCID 0000-0003-4519-1931)
Source: Journal of Research in Science Teaching. 2024 61(1):38-69.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 32
Publication Date: 2024
Sponsoring Agency: National Science Foundation (NSF), Division of Undergraduate Education (DUE)
Contract Number: 1561159
1561150
1561149
1561155
Document Type: Journal Articles
Reports - Research
Education Level: Junior High Schools
Middle Schools
Secondary Education
Descriptors: Middle School Students, Competence, Science Process Skills, Persuasive Discourse, Secondary School Science, Automation, Computer Assisted Testing, Scoring, Prediction, Scores, Models, English Language Learners, Test Bias
Geographic Terms: California (San Francisco)
DOI: 10.1002/tea.21864
ISSN: 0022-4308
1098-2736
Abstract: Argumentation is fundamental to science education, both as a prominent feature of scientific reasoning and as an effective mode of learning--a perspective reflected in contemporary frameworks and standards. The successful implementation of argumentation in school science, however, requires a paradigm shift in science assessment from the measurement of knowledge and understanding to the measurement of performance and knowledge in use. Performance tasks requiring argumentation must capture the many ways students can construct and evaluate arguments in science, yet such tasks are both expensive and resource-intensive to score. In this study we explore how machine learning text classification techniques can be applied to develop efficient, valid, and accurate constructed-response measures of students' competency with written scientific argumentation that are aligned with a validated argumentation learning progression. Data come from 933 middle school students in the San Francisco Bay Area and are based on three sets of argumentation items in three different science contexts. The findings demonstrate that we have been able to develop computer scoring models that can achieve substantial to almost perfect agreement between human-assigned and computer-predicted scores. Model performance was slightly weaker for harder items targeting higher levels of the learning progression, largely due to the linguistic complexity of these responses and the sparsity of higher-level responses in the training data set. Comparing the efficacy of different scoring approaches revealed that breaking down students' arguments into multiple components (e.g., the presence of an accurate claim or providing sufficient evidence), developing computer models for each component, and combining scores from these analytic components into a holistic score produced better results than holistic scoring approaches. However, this analytical approach was found to be differentially biased when scoring responses from English learners (EL) students as compared to responses from non-EL students on some items. Differences in the severity between human and computer scores for EL between these approaches are explored, and potential sources of bias in automated scoring are discussed.
Abstractor: As Provided
Entry Date: 2023
Accession Number: EJ1405261
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1405261
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Using Automated Analysis to Assess Middle School Students' Competence with Scientific Argumentation
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Christopher+D%2E+Wilson%22">Christopher D. Wilson</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-7342-992X">0000-0002-7342-992X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Kevin+C%2E+Haudek%22">Kevin C. Haudek</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1422-6038">0000-0003-1422-6038</externalLink>)<br /><searchLink fieldCode="AR" term="%22Jonathan+F%2E+Osborne%22">Jonathan F. Osborne</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-8096-208X">0000-0001-8096-208X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Zoë+E%2E+Buck+Bracey%22">Zoë E. Buck Bracey</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-4826-3582">0000-0003-4826-3582</externalLink>)<br /><searchLink fieldCode="AR" term="%22Tina+Cheuk%22">Tina Cheuk</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7841-2492">0000-0001-7841-2492</externalLink>)<br /><searchLink fieldCode="AR" term="%22Brian+M%2E+Donovan%22">Brian M. Donovan</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-2329-4198">0000-0003-2329-4198</externalLink>)<br /><searchLink fieldCode="AR" term="%22Molly+A%2E+M%2E+Stuhlsatz%22">Molly A. M. Stuhlsatz</searchLink><br /><searchLink fieldCode="AR" term="%22Marisol+M%2E+Santiago%22">Marisol M. Santiago</searchLink><br /><searchLink fieldCode="AR" term="%22Xiaoming+Zhai%22">Xiaoming Zhai</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-4519-1931">0000-0003-4519-1931</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Research+in+Science+Teaching%22"><i>Journal of Research in Science Teaching</i></searchLink>. 2024 61(1):38-69.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 32
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: National Science Foundation (NSF), Division of Undergraduate Education (DUE)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: 1561159<br />1561150<br />1561149<br />1561155
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Middle+School+Students%22">Middle School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Competence%22">Competence</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Process+Skills%22">Science Process Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Persuasive+Discourse%22">Persuasive Discourse</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Science%22">Secondary School Science</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction%22">Prediction</searchLink><br /><searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22English+Language+Learners%22">English Language Learners</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Bias%22">Test Bias</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22California+%28San+Francisco%29%22">California (San Francisco)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1002/tea.21864
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0022-4308<br />1098-2736
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Argumentation is fundamental to science education, both as a prominent feature of scientific reasoning and as an effective mode of learning--a perspective reflected in contemporary frameworks and standards. The successful implementation of argumentation in school science, however, requires a paradigm shift in science assessment from the measurement of knowledge and understanding to the measurement of performance and knowledge in use. Performance tasks requiring argumentation must capture the many ways students can construct and evaluate arguments in science, yet such tasks are both expensive and resource-intensive to score. In this study we explore how machine learning text classification techniques can be applied to develop efficient, valid, and accurate constructed-response measures of students' competency with written scientific argumentation that are aligned with a validated argumentation learning progression. Data come from 933 middle school students in the San Francisco Bay Area and are based on three sets of argumentation items in three different science contexts. The findings demonstrate that we have been able to develop computer scoring models that can achieve substantial to almost perfect agreement between human-assigned and computer-predicted scores. Model performance was slightly weaker for harder items targeting higher levels of the learning progression, largely due to the linguistic complexity of these responses and the sparsity of higher-level responses in the training data set. Comparing the efficacy of different scoring approaches revealed that breaking down students' arguments into multiple components (e.g., the presence of an accurate claim or providing sufficient evidence), developing computer models for each component, and combining scores from these analytic components into a holistic score produced better results than holistic scoring approaches. However, this analytical approach was found to be differentially biased when scoring responses from English learners (EL) students as compared to responses from non-EL students on some items. Differences in the severity between human and computer scores for EL between these approaches are explored, and potential sources of bias in automated scoring are discussed.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2023
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1405261
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1405261
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1002/tea.21864
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 32
        StartPage: 38
    Subjects:
      – SubjectFull: Middle School Students
        Type: general
      – SubjectFull: Competence
        Type: general
      – SubjectFull: Science Process Skills
        Type: general
      – SubjectFull: Persuasive Discourse
        Type: general
      – SubjectFull: Secondary School Science
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Prediction
        Type: general
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: English Language Learners
        Type: general
      – SubjectFull: Test Bias
        Type: general
      – SubjectFull: California (San Francisco)
        Type: general
    Titles:
      – TitleFull: Using Automated Analysis to Assess Middle School Students' Competence with Scientific Argumentation
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Christopher D. Wilson
      – PersonEntity:
          Name:
            NameFull: Kevin C. Haudek
      – PersonEntity:
          Name:
            NameFull: Jonathan F. Osborne
      – PersonEntity:
          Name:
            NameFull: Zoë E. Buck Bracey
      – PersonEntity:
          Name:
            NameFull: Tina Cheuk
      – PersonEntity:
          Name:
            NameFull: Brian M. Donovan
      – PersonEntity:
          Name:
            NameFull: Molly A. M. Stuhlsatz
      – PersonEntity:
          Name:
            NameFull: Marisol M. Santiago
      – PersonEntity:
          Name:
            NameFull: Xiaoming Zhai
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 0022-4308
            – Type: issn-electronic
              Value: 1098-2736
          Numbering:
            – Type: volume
              Value: 61
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Research in Science Teaching
              Type: main
ResultId 1