View in EDS

Can Alternative Grading Improve Student Interactions in Automatically Graded Programming Assignments?

Saved in:
Bibliographic Details
Title: Can Alternative Grading Improve Student Interactions in Automatically Graded Programming Assignments?
Language: English
Authors: Kyle D. Chin (ORCID 0009-0001-0863-1101), Katharine Kerr (ORCID 0009-0009-9579-175X), Nicholas C. Bradley (ORCID 0000-0001-9974-0794), Reid Holmes (ORCID 0000-0003-4213-494X)
Source: ACM Transactions on Computing Education. 2026 26(1).
Availability: Association for Computing Machinery. 1601 Broadway 10th Floor, New York, NY 10119. Tel: 800-342-6626; Tel: 212-626-0500; Fax: 212-944-1318; e-mail: acmhelp@acm.org; Web site: http://toce.acm.org/
Peer Reviewed: Y
Page Count: 22
Publication Date: 2026
Document Type: Journal Articles
Reports - Research
Education Level: Postsecondary Education
Descriptors: Alternative Assessment, Grading, Automation, Programming, Assignments, Coding, Computer Software, Student Behavior, Feedback (Response), Student Attitudes, Barriers, Postsecondary Education, Active Learning, Student Projects, Computer Science Education
DOI: 10.1145/3767736
ISSN: 1946-6226
Abstract: Background: Automated assessments, often implemented as test-based autograders, are widely used in the form of hidden oracles, providing automated formative feedback to students on their code solutions. While autograders provide some educational benefits--particularly around efficient scaling of grading--they have been shown to encourage adverse student behaviors that hinder learning. For example, students try to debug their solutions into existence through trial-and-error debugging in pursuit of maximum points, without the valuable, careful introspection on their work. Objectives: This study investigates how a grade scale affects students' software development behaviors in response to automated feedback. In contrast to an autograder with an oracle suite of test cases, industrial developers "do not" have access to an oracle test suite that can tell them what cases their code handles or mishandles. Developers must instead rely on careful reasoning to co-evolve their test code with their product code to produce and maintain high-quality software. We hypothesize that alternative grading can be an effective tool to influence students to practice more reflection during development while maintaining the infrastructural and pedagogical benefits of automated assessments. Methods: We deployed a coarse-grained, alternative grading approach--bucket grading--which assessed solutions to be in one of only four bucket grades, representing a high-level assessment of the student's project quality, to a third-year post-secondary, project-based software engineering class with 300+ students. This study uses a mixed qualitative and quantitative methodology to compare a previous offering of the course that used a traditional, points-based grading scheme against the bucket grading offering. Findings: We find that coarse-grained formative feedback via bucket grading improves student-autograder interaction: students wrote stronger, more focused test suites because they reflected more deeply before making changes. Students were appreciative of bucket grading since it provided them leniency in their grade and greater clarity on the overall quality of their solutions. Ultimately, we will continue to use this approach going forward since incorporating bucket grading meaningfully improved both the staff and student autograder experiences.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1497195
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1497195
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Can Alternative Grading Improve Student Interactions in Automatically Graded Programming Assignments?
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Kyle+D%2E+Chin%22">Kyle D. Chin</searchLink> (ORCID <externalLink term="https://orcid.org/0009-0001-0863-1101">0009-0001-0863-1101</externalLink>)<br /><searchLink fieldCode="AR" term="%22Katharine+Kerr%22">Katharine Kerr</searchLink> (ORCID <externalLink term="https://orcid.org/0009-0009-9579-175X">0009-0009-9579-175X</externalLink>)<br /><searchLink fieldCode="AR" term="%22Nicholas+C%2E+Bradley%22">Nicholas C. Bradley</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9974-0794">0000-0001-9974-0794</externalLink>)<br /><searchLink fieldCode="AR" term="%22Reid+Holmes%22">Reid Holmes</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-4213-494X">0000-0003-4213-494X</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22ACM+Transactions+on+Computing+Education%22"><i>ACM Transactions on Computing Education</i></searchLink>. 2026 26(1).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Association for Computing Machinery. 1601 Broadway 10th Floor, New York, NY 10119. Tel: 800-342-6626; Tel: 212-626-0500; Fax: 212-944-1318; e-mail: acmhelp@acm.org; Web site: http://toce.acm.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 22
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Alternative+Assessment%22">Alternative Assessment</searchLink><br /><searchLink fieldCode="DE" term="%22Grading%22">Grading</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Programming%22">Programming</searchLink><br /><searchLink fieldCode="DE" term="%22Assignments%22">Assignments</searchLink><br /><searchLink fieldCode="DE" term="%22Coding%22">Coding</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Software%22">Computer Software</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Behavior%22">Student Behavior</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Attitudes%22">Student Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Barriers%22">Barriers</searchLink><br /><searchLink fieldCode="DE" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Active+Learning%22">Active Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Projects%22">Student Projects</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Science+Education%22">Computer Science Education</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1145/3767736
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1946-6226
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: Automated assessments, often implemented as test-based autograders, are widely used in the form of hidden oracles, providing automated formative feedback to students on their code solutions. While autograders provide some educational benefits--particularly around efficient scaling of grading--they have been shown to encourage adverse student behaviors that hinder learning. For example, students try to debug their solutions into existence through trial-and-error debugging in pursuit of maximum points, without the valuable, careful introspection on their work. Objectives: This study investigates how a grade scale affects students' software development behaviors in response to automated feedback. In contrast to an autograder with an oracle suite of test cases, industrial developers "do not" have access to an oracle test suite that can tell them what cases their code handles or mishandles. Developers must instead rely on careful reasoning to co-evolve their test code with their product code to produce and maintain high-quality software. We hypothesize that alternative grading can be an effective tool to influence students to practice more reflection during development while maintaining the infrastructural and pedagogical benefits of automated assessments. Methods: We deployed a coarse-grained, alternative grading approach--bucket grading--which assessed solutions to be in one of only four bucket grades, representing a high-level assessment of the student's project quality, to a third-year post-secondary, project-based software engineering class with 300+ students. This study uses a mixed qualitative and quantitative methodology to compare a previous offering of the course that used a traditional, points-based grading scheme against the bucket grading offering. Findings: We find that coarse-grained formative feedback via bucket grading improves student-autograder interaction: students wrote stronger, more focused test suites because they reflected more deeply before making changes. Students were appreciative of bucket grading since it provided them leniency in their grade and greater clarity on the overall quality of their solutions. Ultimately, we will continue to use this approach going forward since incorporating bucket grading meaningfully improved both the staff and student autograder experiences.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1497195
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1497195
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1145/3767736
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 22
    Subjects:
      – SubjectFull: Alternative Assessment
        Type: general
      – SubjectFull: Grading
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Programming
        Type: general
      – SubjectFull: Assignments
        Type: general
      – SubjectFull: Coding
        Type: general
      – SubjectFull: Computer Software
        Type: general
      – SubjectFull: Student Behavior
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: Student Attitudes
        Type: general
      – SubjectFull: Barriers
        Type: general
      – SubjectFull: Postsecondary Education
        Type: general
      – SubjectFull: Active Learning
        Type: general
      – SubjectFull: Student Projects
        Type: general
      – SubjectFull: Computer Science Education
        Type: general
    Titles:
      – TitleFull: Can Alternative Grading Improve Student Interactions in Automatically Graded Programming Assignments?
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Kyle D. Chin
      – PersonEntity:
          Name:
            NameFull: Katharine Kerr
      – PersonEntity:
          Name:
            NameFull: Nicholas C. Bradley
      – PersonEntity:
          Name:
            NameFull: Reid Holmes
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 1946-6226
          Numbering:
            – Type: volume
              Value: 26
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: ACM Transactions on Computing Education
              Type: main
ResultId 1