View in EDS

Examining the Accuracy of a Conversation-Based Assessment in Interpreting English Learners' Written Responses. Research Report. ETS RR-21-03

Saved in:
Bibliographic Details
Title: Examining the Accuracy of a Conversation-Based Assessment in Interpreting English Learners' Written Responses. Research Report. ETS RR-21-03
Language: English
Authors: Lopez, Alexis A., Guzman-Orth, Danielle, Zapata-Rivera, Diego, Forsyth, Carolyn M., Luce, Christine
Source: ETS Research Report Series. Dec 2021.
Availability: Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Peer Reviewed: Y
Page Count: 15
Publication Date: 2021
Document Type: Journal Articles
Reports - Research
Education Level: Junior High Schools
Middle Schools
Secondary Education
Descriptors: Accuracy, English Language Learners, Language Proficiency, Language Tests, Communicative Competence (Languages), Test Format, Mathematics Tests, Middle School Students, Task Analysis, Computer Assisted Testing, Peer Relationship, Teacher Student Relationship, Written Language, Responses, Test Items, Item Analysis, English (Second Language), Second Language Learning
ISSN: 2330-8516
Abstract: Substantial progress has been made toward applying technology enhanced conversation-based assessments (CBAs) to measure the English-language proficiency of English learners (ELs). CBAs are conversation-based systems that use conversations among computer-animated agents and a test taker. We expanded the design and capability of prior conversation-based instructional and assessment systems and developed a CBA designed to measure the English language skills and the mathematics knowledge of middle school ELs. The prototype CBA simulates an authentic and engaging mathematics classroom where the test taker interacts with two virtual agents to solve math problems. We embedded feedback and supports that are triggered by how the CBA interprets students' written responses. In this study, we administered the CBA to middle school ELs (N = 82) residing in the United States. We examined the extent to which the CBA system was able to consistently interpret the students' responses (722 responses for the 82 students). The study findings helped us to understand the factors that affect the accuracy of the CBA system's interpretations and shed light on how to improve CBA systems that incorporate scaffolding.
Abstractor: As Provided
Entry Date: 2022
Accession Number: EJ1340994
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1340994
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1340994
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Examining the Accuracy of a Conversation-Based Assessment in Interpreting English Learners' Written Responses. Research Report. ETS RR-21-03
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Lopez%2C+Alexis+A%2E%22">Lopez, Alexis A.</searchLink><br /><searchLink fieldCode="AR" term="%22Guzman-Orth%2C+Danielle%22">Guzman-Orth, Danielle</searchLink><br /><searchLink fieldCode="AR" term="%22Zapata-Rivera%2C+Diego%22">Zapata-Rivera, Diego</searchLink><br /><searchLink fieldCode="AR" term="%22Forsyth%2C+Carolyn+M%2E%22">Forsyth, Carolyn M.</searchLink><br /><searchLink fieldCode="AR" term="%22Luce%2C+Christine%22">Luce, Christine</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22ETS+Research+Report+Series%22"><i>ETS Research Report Series</i></searchLink>. Dec 2021.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 15
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2021
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22English+Language+Learners%22">English Language Learners</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Proficiency%22">Language Proficiency</searchLink><br /><searchLink fieldCode="DE" term="%22Language+Tests%22">Language Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Communicative+Competence+%28Languages%29%22">Communicative Competence (Languages)</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Format%22">Test Format</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Tests%22">Mathematics Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Students%22">Middle School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Task+Analysis%22">Task Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Peer+Relationship%22">Peer Relationship</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Student+Relationship%22">Teacher Student Relationship</searchLink><br /><searchLink fieldCode="DE" term="%22Written+Language%22">Written Language</searchLink><br /><searchLink fieldCode="DE" term="%22Responses%22">Responses</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Items%22">Test Items</searchLink><br /><searchLink fieldCode="DE" term="%22Item+Analysis%22">Item Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22English+%28Second+Language%29%22">English (Second Language)</searchLink><br /><searchLink fieldCode="DE" term="%22Second+Language+Learning%22">Second Language Learning</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2330-8516
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Substantial progress has been made toward applying technology enhanced conversation-based assessments (CBAs) to measure the English-language proficiency of English learners (ELs). CBAs are conversation-based systems that use conversations among computer-animated agents and a test taker. We expanded the design and capability of prior conversation-based instructional and assessment systems and developed a CBA designed to measure the English language skills and the mathematics knowledge of middle school ELs. The prototype CBA simulates an authentic and engaging mathematics classroom where the test taker interacts with two virtual agents to solve math problems. We embedded feedback and supports that are triggered by how the CBA interprets students' written responses. In this study, we administered the CBA to middle school ELs (N = 82) residing in the United States. We examined the extent to which the CBA system was able to consistently interpret the students' responses (722 responses for the 82 students). The study findings helped us to understand the factors that affect the accuracy of the CBA system's interpretations and shed light on how to improve CBA systems that incorporate scaffolding.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2022
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1340994
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1340994
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 15
    Subjects:
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: English Language Learners
        Type: general
      – SubjectFull: Language Proficiency
        Type: general
      – SubjectFull: Language Tests
        Type: general
      – SubjectFull: Communicative Competence (Languages)
        Type: general
      – SubjectFull: Test Format
        Type: general
      – SubjectFull: Mathematics Tests
        Type: general
      – SubjectFull: Middle School Students
        Type: general
      – SubjectFull: Task Analysis
        Type: general
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Peer Relationship
        Type: general
      – SubjectFull: Teacher Student Relationship
        Type: general
      – SubjectFull: Written Language
        Type: general
      – SubjectFull: Responses
        Type: general
      – SubjectFull: Test Items
        Type: general
      – SubjectFull: Item Analysis
        Type: general
      – SubjectFull: English (Second Language)
        Type: general
      – SubjectFull: Second Language Learning
        Type: general
    Titles:
      – TitleFull: Examining the Accuracy of a Conversation-Based Assessment in Interpreting English Learners' Written Responses. Research Report. ETS RR-21-03
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Lopez, Alexis A.
      – PersonEntity:
          Name:
            NameFull: Guzman-Orth, Danielle
      – PersonEntity:
          Name:
            NameFull: Zapata-Rivera, Diego
      – PersonEntity:
          Name:
            NameFull: Forsyth, Carolyn M.
      – PersonEntity:
          Name:
            NameFull: Luce, Christine
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2021
          Identifiers:
            – Type: issn-electronic
              Value: 2330-8516
          Titles:
            – TitleFull: ETS Research Report Series
              Type: main
ResultId 1