View in EDS

Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers

Saved in:
Bibliographic Details
Title: Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers
Language: English
Authors: Neil C. C. Brown (ORCID 0000-0001-6086-2479), Pierre Weill-Tessier (ORCID 0000-0002-6999-8767), Juho Leinonen (ORCID 0000-0001-6829-9449), Paul Denny (ORCID 0000-0002-5150-9806), Michael Kölling (ORCID 0000-0003-0544-2003)
Source: ACM Transactions on Computing Education. 2025 25(3).
Availability: Association for Computing Machinery. 1601 Broadway 10th Floor, New York, NY 10119. Tel: 800-342-6626; Tel: 212-626-0500; Fax: 212-944-1318; e-mail: acmhelp@acm.org; Web site: http://toce.acm.org/
Peer Reviewed: Y
Page Count: 43
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Artificial Intelligence, Programming, Novices, Technology Uses in Education, Cues, Evaluation, Teacher Attitudes, Computer Science Education
DOI: 10.1145/3737885
ISSN: 1946-6226
Abstract: Motivation: Students learning to program often reach states where they are stuck and can make no forward progress--but this may be outside the classroom where no instructor is available to help. In this situation, an automatically generated next-step hint can help them make forward progress and support their learning. It is important to know what makes a good hint or a bad hint, and how to generate good hints automatically in novice programming tools, for example, using large language models (LLMs). Method and Participants: We recruited 44 Java educators from around the world to participate in an online study. We used a set of real student code states as hint-generation scenarios. Participants used a technique known as comparative judgement to rank a set of candidate next-step Java hints, which were generated by LLMs and by five human experienced educators. Participants ranked the hints without being told how they were generated. The hints were generated with no explicit detail given to the LLMs/humans on what the target task was. Participants then filled in a survey with follow-up questions. The ranks of the hints were analysed against a set of extracted hint characteristics using a random forest approach. Findings: We found that LLMs had considerable variation in generating high quality next-step hints for programming novices, with GPT-4 outperforming other models tested. When used with a well-designed prompt, GPT-4 outperformed human experts in generating pedagogically valuable hints. A multi-stage prompt was the most effective LLM prompt. According to a fitted random forest model, the two most important factors of a good hint were length (80-160 words being best) and reading level (US grade nine or below being best). Offering alternative approaches to solving the problem was considered bad, and we found no effect of sentiment. Conclusions: Automatic generation of these hints is immediately viable, given that LLMs outperformed humans--even when the students' task is unknown. Hint length and reading level were more important than several pedagogical features of hints. The fact that it took a group of experts several rounds of experimentation and refinement to design a prompt that achieves this outcome suggests that students on their own are unlikely to be able to produce the same benefit. The prompting task, therefore, should be embedded in an expert-designed tool.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1485290
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1485290
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Neil+C%2E+C%2E+Brown%22">Neil C. C. Brown</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-6086-2479">0000-0001-6086-2479</externalLink>)<br /><searchLink fieldCode="AR" term="%22Pierre+Weill-Tessier%22">Pierre Weill-Tessier</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-6999-8767">0000-0002-6999-8767</externalLink>)<br /><searchLink fieldCode="AR" term="%22Juho+Leinonen%22">Juho Leinonen</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-6829-9449">0000-0001-6829-9449</externalLink>)<br /><searchLink fieldCode="AR" term="%22Paul+Denny%22">Paul Denny</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-5150-9806">0000-0002-5150-9806</externalLink>)<br /><searchLink fieldCode="AR" term="%22Michael+Kölling%22">Michael Kölling</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-0544-2003">0000-0003-0544-2003</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22ACM+Transactions+on+Computing+Education%22"><i>ACM Transactions on Computing Education</i></searchLink>. 2025 25(3).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Association for Computing Machinery. 1601 Broadway 10th Floor, New York, NY 10119. Tel: 800-342-6626; Tel: 212-626-0500; Fax: 212-944-1318; e-mail: acmhelp@acm.org; Web site: http://toce.acm.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 43
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Programming%22">Programming</searchLink><br /><searchLink fieldCode="DE" term="%22Novices%22">Novices</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Cues%22">Cues</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation%22">Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Attitudes%22">Teacher Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Science+Education%22">Computer Science Education</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1145/3737885
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1946-6226
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Motivation: Students learning to program often reach states where they are stuck and can make no forward progress--but this may be outside the classroom where no instructor is available to help. In this situation, an automatically generated next-step hint can help them make forward progress and support their learning. It is important to know what makes a good hint or a bad hint, and how to generate good hints automatically in novice programming tools, for example, using large language models (LLMs). Method and Participants: We recruited 44 Java educators from around the world to participate in an online study. We used a set of real student code states as hint-generation scenarios. Participants used a technique known as comparative judgement to rank a set of candidate next-step Java hints, which were generated by LLMs and by five human experienced educators. Participants ranked the hints without being told how they were generated. The hints were generated with no explicit detail given to the LLMs/humans on what the target task was. Participants then filled in a survey with follow-up questions. The ranks of the hints were analysed against a set of extracted hint characteristics using a random forest approach. Findings: We found that LLMs had considerable variation in generating high quality next-step hints for programming novices, with GPT-4 outperforming other models tested. When used with a well-designed prompt, GPT-4 outperformed human experts in generating pedagogically valuable hints. A multi-stage prompt was the most effective LLM prompt. According to a fitted random forest model, the two most important factors of a good hint were length (80-160 words being best) and reading level (US grade nine or below being best). Offering alternative approaches to solving the problem was considered bad, and we found no effect of sentiment. Conclusions: Automatic generation of these hints is immediately viable, given that LLMs outperformed humans--even when the students' task is unknown. Hint length and reading level were more important than several pedagogical features of hints. The fact that it took a group of experts several rounds of experimentation and refinement to design a prompt that achieves this outcome suggests that students on their own are unlikely to be able to produce the same benefit. The prompting task, therefore, should be embedded in an expert-designed tool.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1485290
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1485290
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1145/3737885
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 43
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Programming
        Type: general
      – SubjectFull: Novices
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Cues
        Type: general
      – SubjectFull: Evaluation
        Type: general
      – SubjectFull: Teacher Attitudes
        Type: general
      – SubjectFull: Computer Science Education
        Type: general
    Titles:
      – TitleFull: Howzat? Appealing to Expert Judgement for Evaluating Human and AI Next-Step Hints for Novice Programmers
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Neil C. C. Brown
      – PersonEntity:
          Name:
            NameFull: Pierre Weill-Tessier
      – PersonEntity:
          Name:
            NameFull: Juho Leinonen
      – PersonEntity:
          Name:
            NameFull: Paul Denny
      – PersonEntity:
          Name:
            NameFull: Michael Kölling
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-electronic
              Value: 1946-6226
          Numbering:
            – Type: volume
              Value: 25
            – Type: issue
              Value: 3
          Titles:
            – TitleFull: ACM Transactions on Computing Education
              Type: main
ResultId 1