View in EDS HTML Full Text PDF Full Text

Digital Module 18: Automated Scoring

Saved in:
Bibliographic Details
Title: Digital Module 18: Automated Scoring
Language: English
Authors: Lottridge, Sue, Burkhardt, Amy, Boyer, Michelle
Source: Educational Measurement: Issues and Practice. Fall 2020 39(3):141-142.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 2
Publication Date: 2020
Document Type: Journal Articles
Reports - Descriptive
Descriptors: Computer Assisted Testing, Scoring, Automation, Educational Assessment, Educational Benefits, Psychometrics, Data Collection, Test Validity, Error of Measurement
DOI: 10.1111/emip.12388
ISSN: 0731-1745
Abstract: In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open-ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows scores to be returned faster at lower cost. In the module, they discuss automated scoring from a number of perspectives. First, they discuss benefits and weaknesses of automated scoring, and what psychometricians should know about automated scoring. Next, they describe the overall process of automated scoring, moving from data collection to engine training to operational scoring. Then, they describe how automated scoring systems work, including the basic functions around score prediction as well as other flagging methods. Finally, they conclude with a discussion of the specific validity demands around automated scoring and how they align with the larger validity demands around test scores. Two data activities are provided. The first is an interactive activity that allows the user to train and evaluate a simple automated scoring engine. The second is a worked example that examines the impact of rater error on test scores. The digital module contains a link to an interactive web application as well as its R-Shiny code, diagnostic quiz questions, activities, curated resources, and a glossary.
Abstractor: As Provided
Notes: https://ncme.elevate.commpartners.com
Entry Date: 2020
Accession Number: EJ1266816
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwE4i6c0AJ_-Bhhy3UkwsNp1AAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDI7ipb5aiSboHWYFZAIBEICBmxfLYNEU4UGAJHHpMD4QVwY2ge9RCJErv2dh0Ry1YKwr0oy8LlLxbs0O1Xc8qVisFn5EYpzQQN3sTIOEAfp3QzS2NvC_4tWSvYeB3Ju8vVohBh9S_uL6Af4M6A-LfxMLJLPOB3Oyt5dP1dyhv7Daw05jaV6BYXmHA68NpDAeGvXZA7qh58gsluzGFXezjEjCq_qZ_G0tUqWqMOGS
Text:
  Availability: 1
  Value: <anid>AN0145698234;ems01sep.20;2025Apr02.13:03;v2.2.500</anid> <title id="AN0145698234-1">Digital Module 18: Automated Scoring https://ncme.elevate.commpartners.com </title> <p>In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open‐ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows scores to be returned faster at lower cost. In the module, they discuss automated scoring from a number of perspectives. First, they discuss benefits and weaknesses of automated scoring, and what psychometricians should know about automated scoring. Next, they describe the overall process of automated scoring, moving from data collection to engine training to operational scoring. Then, they describe how automated scoring systems work, including the basic functions around score prediction as well as other flagging methods. Finally, they conclude with a discussion of the specific validity demands around automated scoring and how they align with the larger validity demands around test scores. Two data activities are provided. The first is an interactive activity that allows the user to train and evaluate a simple automated scoring engine. The second is a worked example that examines the impact of rater error on test scores. The digital module contains a link to an interactive web application as well as its R‐Shiny code, diagnostic quiz questions, activities, curated resources, and a glossary.</p> <p>Keywords: automated scoring; constructed response items; hand‐scoring; machine learning; natural language processes</p> <hd id="AN0145698234-2">Prerequisite Knowledge</hd> <p>The intended audience is psychometricians who range in experience and areas of specialization and know very little about automated scoring. To get the most out of the module, it might be beneficial to have a basic understanding of the areas below to fully appreciate how automated scoring works and how it fits into the psychometric workflow:</p> <p></p> <ulist> <item> Industry standards for validity and reliability (e.g., as described in the AERA, APA, and NCME standards).</item> <p></p> <item> Foundational statistical concepts (e.g., means, variation, correlations).</item> <p></p> <item> Development processes for open‐ended test items.</item> <p></p> <item> Human scoring processes (e.g., rubric development) and quality‐control metrics (e.g., reader agreement indices).</item> </ulist> <p>Reading the following NCME ITEMS print modules may serve as a useful introduction to the prerequisite knowledge:</p> <p></p> <ulist> <item> Module 1: Performance Assessments: Design and Development.</item> <p></p> <item> Module 15: Assessing Student Achievement with Term Papers and Written Reports.</item> <p></p> <item> Module 3: Reliability of Scores from Teacher‐Made Tests.</item> <p></p> <item> Module 43: Data Mining for Classification and Regression.</item> <p></p> <item> Module 42: Simulation Studies in Psychometrics.</item> </ulist> <p>These modules and others are available for free in the ITEMS portal.</p> <hd id="AN0145698234-3">Learning Objectives</hd> <p>Upon completion of this module, learners should be able to:</p> <p></p> <ulist> <item> understand how their analytic work might intersect with automated scoring,</item> <p></p> <item> know that there are different scoring sources (i.e., human, machine),</item> <p></p> <item> identify the decision points around automated scoring that psychometricians make,</item> <p></p> <item> understand how automated scores are produced,</item> <p></p> <item> know the key dimensions of an automated scoring engine validation program,</item> <p></p> <item> examine the impact of rater error on common agreement metrics.</item> </ulist> <hd id="AN0145698234-4">Module Structure</hd> <p>The digital module is divided into the following sections, which can be reviewed sequentially or independently [<emph>approximate completion times in parentheses</emph>].</p> <p></p> <ulist> <item> ‐ Module Introduction [ <emph>5 minutes</emph> ].</item> <p></p> <item> ‐ Section 1: Conceptual Foundations [ <emph>25 minutes</emph> ].</item> <p></p> <item> ‐ Section 2: Overview of Automated Scoring Processes [ <emph>25 minutes</emph> ].</item> <p></p> <item> ‐ Section 3: How Automated Scoring Works [ <emph>25 minutes</emph> ].</item> <p></p> <item> ‐ Section 4: Developing a Program of Validation [ <emph>25 minutes</emph> ].</item> <p></p> <item> ‐ Section 5: Data Activity [ <emph>20 minutes</emph> ],</item> <p></p> <item> ‐ Section 6: Worked Example [ <emph>15 minutes</emph> ],</item> <p></p> <item> ‐ Section 7: Quizzes [ <emph>20 minutes</emph> ].</item> </ulist> <p>In the portal site, there is a video version of the core content as well as a handout with all core slides along with other materials.</p> <hd id="AN0145698234-5">Instructors</hd> <p>GRAPH</p> <p> <bold>Susan Lottridge</bold>, Cambium Assessment, Inc. Sue Lottridge, Ph.D. is a Senior Director of Automated Scoring at the Cambium Assessment, Inc. (CAI). In this role, she leads CAI's machine learning and scoring team on the research, development, and operation of CAI's automated scoring software. This software includes automated essay scoring, short answer scoring, automated speech scoring, and an engine that detects disturbing content in student responses. Dr. Lottridge has worked in automated scoring for 12 years and has contributed to the design, research, and use of multiple automated scoring engines including equation scoring, essay scoring, short answer scoring, alert detection, and dialogue systems. She earned her Ph.D. from James Madison University in assessment and measurement (2006) and holds master's degrees in Mathematics and in Computer Science from the University of Wisconsin‐Madison (1997).</p> <p>GRAPH</p> <p> <bold>Amy Burkhardt</bold>, University of Colorado – Boulder. Amy Burkhardt is a PhD Candidate in Research and Evaluation Methodology with an emphasis in Human Language Technology at the University of Colorado, Boulder. She has been involved in the development of two automated scoring systems. Ongoing research projects include the automatic detection of students reporting harm within online tests, the use of machine learning to explore public discourse around educational policies, and considerations in psychometric modeling when making diagnostic inferences aligned to a learning progression.</p> <p>GRAPH</p> <p> <bold>Michelle Boyer</bold>, Center for Assessment. Michelle Boyer, Ph.D. is a Senior Associate at The National Center for the Improvement of Educational Assessment, Inc. Dr. Boyer consults with states and organizations on such issues as assessment systems, validity of score interpretations, scoring design and evaluation criteria for both human and automated scoring, assessment literacy, and score comparability. She is also a regular contributor to professional publications and the annual conferences of AERA, NCME, and CCSSO. Her most recent research focuses on evaluating the quality of automated scoring and its impact test score scales and test equating solutions. Dr. Boyer earned her Ph.D. from the University of Massachusetts, Amherst in Research, Educational Measurement, and Psychometrics (2018).</p> <hd id="AN0145698234-6">Instructional Designer</hd> <p>GRAPH</p> <p> <bold>André A. Rupp</bold>, is the co‐author and co‐editor of two award‐winning interdisciplinary books entitled <emph>Diagnostic Measurement: Theory, Methods, and Applications</emph> (2010) and <emph>The Handbook of Cognition and Assessment: Frameworks, Methodologies, and Applications</emph> (2016) and has just published the <emph>Handbook of Automated Scoring: Theory into Practice</emph> (2020). His research synthesis‐ and framework‐oriented work has appeared in a wide variety of prestigious peer‐reviewed journals. Among other things, he is passionate about improving processes for interdisciplinary collaborations during the development and implementation of scoring solutions for digitally delivered assessments. Consequently, he is very excited to serve as the associate editor/lead instructional designer of the ITEMS portal for NCME whose mission is to provide free digital resources to support self‐directed learning and professional development.</p> <ref id="AN0145698234-7"> <title> Footnotes </title> <blist> <bibl id="bib1" type="bt">1</bibl> <bibtext> Available in the ITEMS Portal at https://ncme.elevate.commpartners.com</bibtext> </blist> </ref> <aug> <p>By Sue Lottridge; Amy Burkhardt and Michelle Boyer</p> <p>Reported by Author; Author; Author</p> </aug>
Header DbId: eric
DbLabel: ERIC
An: EJ1266816
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Digital Module 18: Automated Scoring
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Lottridge%2C+Sue%22">Lottridge, Sue</searchLink><br /><searchLink fieldCode="AR" term="%22Burkhardt%2C+Amy%22">Burkhardt, Amy</searchLink><br /><searchLink fieldCode="AR" term="%22Boyer%2C+Michelle%22">Boyer, Michelle</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Measurement%3A+Issues+and+Practice%22"><i>Educational Measurement: Issues and Practice</i></searchLink>. Fall 2020 39(3):141-142.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 2
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2020
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Descriptive
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Scoring%22">Scoring</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Assessment%22">Educational Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Benefits%22">Educational Benefits</searchLink><br /><searchLink fieldCode="DE" term="%22Psychometrics%22">Psychometrics</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Collection%22">Data Collection</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Validity%22">Test Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Error+of+Measurement%22">Error of Measurement</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/emip.12388
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0731-1745
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In this digital ITEMS module, Dr. Sue Lottridge, Amy Burkhardt, and Dr. Michelle Boyer provide an overview of automated scoring. Automated scoring is the use of computer algorithms to score unconstrained open-ended test items by mimicking human scoring. The use of automated scoring is increasing in educational assessment programs because it allows scores to be returned faster at lower cost. In the module, they discuss automated scoring from a number of perspectives. First, they discuss benefits and weaknesses of automated scoring, and what psychometricians should know about automated scoring. Next, they describe the overall process of automated scoring, moving from data collection to engine training to operational scoring. Then, they describe how automated scoring systems work, including the basic functions around score prediction as well as other flagging methods. Finally, they conclude with a discussion of the specific validity demands around automated scoring and how they align with the larger validity demands around test scores. Two data activities are provided. The first is an interactive activity that allows the user to train and evaluate a simple automated scoring engine. The second is a worked example that examines the impact of rater error on test scores. The digital module contains a link to an interactive web application as well as its R-Shiny code, diagnostic quiz questions, activities, curated resources, and a glossary.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://ncme.elevate.commpartners.com
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2020
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1266816
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1266816
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/emip.12388
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 2
        StartPage: 141
    Subjects:
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Scoring
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Educational Assessment
        Type: general
      – SubjectFull: Educational Benefits
        Type: general
      – SubjectFull: Psychometrics
        Type: general
      – SubjectFull: Data Collection
        Type: general
      – SubjectFull: Test Validity
        Type: general
      – SubjectFull: Error of Measurement
        Type: general
    Titles:
      – TitleFull: Digital Module 18: Automated Scoring
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Lottridge, Sue
      – PersonEntity:
          Name:
            NameFull: Burkhardt, Amy
      – PersonEntity:
          Name:
            NameFull: Boyer, Michelle
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2020
          Identifiers:
            – Type: issn-print
              Value: 0731-1745
          Numbering:
            – Type: volume
              Value: 39
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: Educational Measurement: Issues and Practice
              Type: main
ResultId 1