View in EDS

Do AI Chatbots Improve Students' Learning Performance in Programming Education? Evidence from a Meta-Analysis

Saved in:
Bibliographic Details
Title: Do AI Chatbots Improve Students' Learning Performance in Programming Education? Evidence from a Meta-Analysis
Language: English
Authors: Hongji Deng (ORCID 0009-0009-4597-8188), Hui Chen (ORCID 0000-0002-1492-9385), Yan Dong (ORCID 0000-0003-1678-6370)
Source: Journal of Educational Computing Research. 2026 64(5):1323-1359.
Availability: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Peer Reviewed: Y
Page Count: 37
Publication Date: 2026
Document Type: Journal Articles
Information Analyses
Education Level: Elementary Education
Secondary Education
Postsecondary Education
Descriptors: Artificial Intelligence, Synchronous Communication, Programming, Academic Achievement, Technology Uses in Education, Elementary Education, Secondary Education, Postsecondary Education, Time Factors (Learning), Educational Environment, Research Design, Programming Languages, Teaching Methods, Influences
DOI: 10.1177/07356331261424211
ISSN: 0735-6331
1541-4140
Abstract: AI chatbots have emerged as innovative educational tools and drawn increasing attention from educators and researchers in programming education. Although previous research has highlighted potentials of applying AI chatbots in programming education, there is a lack of empirical evidence to understand the overall effects of using AI chatbots in programming learning as well as the critical factors that influence the effects. To fill this gap, this study conducted a meta-analysis of 32 empirical studies published between 2015 and 2025 to examine the overall effect size of applying AI chatbots on programming learning performance and identify significant moderators. The results indicated a small-to-medium effect on posttest performance (g+ = 0.538, 95% CI [0.202, 0.873], p < 0.01) and a medium-to-large effect on practice performance (g+ = 0.650, 95% CI [0.330, 0.970], p < 0.001), based on robust variance estimation models. Moderator analyses revealed that research design and AI chatbot-to-student ratio significantly influenced posttest performance. Specifically, true experimental designs demonstrated significantly larger effects than quasi-experimental designs, and a 1:1 chatbot-student ratio was substantially more effective than a 1:N ratio. These findings underscore the potential of AI chatbots in programming education and offer practical insights for optimizing their integration into instructional design.
Abstractor: As Provided
Notes: https://osf.io/gnk6d
Entry Date: 2026
Accession Number: EJ1506771
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1506771
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Do AI Chatbots Improve Students&#39; Learning Performance in Programming Education? Evidence from a Meta-Analysis
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: &lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Hongji+Deng%22&quot;&gt;Hongji Deng&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0009-0009-4597-8188&quot;&gt;0009-0009-4597-8188&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Hui+Chen%22&quot;&gt;Hui Chen&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0002-1492-9385&quot;&gt;0000-0002-1492-9385&lt;/externalLink&gt;)&lt;br /&gt;&lt;searchLink fieldCode=&quot;AR&quot; term=&quot;%22Yan+Dong%22&quot;&gt;Yan Dong&lt;/searchLink&gt; (ORCID &lt;externalLink term=&quot;https://orcid.org/0000-0003-1678-6370&quot;&gt;0000-0003-1678-6370&lt;/externalLink&gt;)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: &lt;searchLink fieldCode=&quot;SO&quot; term=&quot;%22Journal+of+Educational+Computing+Research%22&quot;&gt;&lt;i&gt;Journal of Educational Computing Research&lt;/i&gt;&lt;/searchLink&gt;. 2026 64(5):1323-1359.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 37
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles&lt;br /&gt;Information Analyses
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: &lt;searchLink fieldCode=&quot;EL&quot; term=&quot;%22Elementary+Education%22&quot;&gt;Elementary Education&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;EL&quot; term=&quot;%22Secondary+Education%22&quot;&gt;Secondary Education&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;EL&quot; term=&quot;%22Postsecondary+Education%22&quot;&gt;Postsecondary Education&lt;/searchLink&gt;
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: &lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Artificial+Intelligence%22&quot;&gt;Artificial Intelligence&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Synchronous+Communication%22&quot;&gt;Synchronous Communication&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Programming%22&quot;&gt;Programming&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Academic+Achievement%22&quot;&gt;Academic Achievement&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Technology+Uses+in+Education%22&quot;&gt;Technology Uses in Education&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Elementary+Education%22&quot;&gt;Elementary Education&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Secondary+Education%22&quot;&gt;Secondary Education&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Postsecondary+Education%22&quot;&gt;Postsecondary Education&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Time+Factors+%28Learning%29%22&quot;&gt;Time Factors (Learning)&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Educational+Environment%22&quot;&gt;Educational Environment&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Research+Design%22&quot;&gt;Research Design&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Programming+Languages%22&quot;&gt;Programming Languages&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Teaching+Methods%22&quot;&gt;Teaching Methods&lt;/searchLink&gt;&lt;br /&gt;&lt;searchLink fieldCode=&quot;DE&quot; term=&quot;%22Influences%22&quot;&gt;Influences&lt;/searchLink&gt;
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1177/07356331261424211
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0735-6331&lt;br /&gt;1541-4140
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: AI chatbots have emerged as innovative educational tools and drawn increasing attention from educators and researchers in programming education. Although previous research has highlighted potentials of applying AI chatbots in programming education, there is a lack of empirical evidence to understand the overall effects of using AI chatbots in programming learning as well as the critical factors that influence the effects. To fill this gap, this study conducted a meta-analysis of 32 empirical studies published between 2015 and 2025 to examine the overall effect size of applying AI chatbots on programming learning performance and identify significant moderators. The results indicated a small-to-medium effect on posttest performance (g+ = 0.538, 95% CI [0.202, 0.873], p &lt; 0.01) and a medium-to-large effect on practice performance (g+ = 0.650, 95% CI [0.330, 0.970], p &lt; 0.001), based on robust variance estimation models. Moderator analyses revealed that research design and AI chatbot-to-student ratio significantly influenced posttest performance. Specifically, true experimental designs demonstrated significantly larger effects than quasi-experimental designs, and a 1:1 chatbot-student ratio was substantially more effective than a 1:N ratio. These findings underscore the potential of AI chatbots in programming education and offer practical insights for optimizing their integration into instructional design.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://osf.io/gnk6d
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1506771
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1506771
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1177/07356331261424211
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 37
        StartPage: 1323
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Synchronous Communication
        Type: general
      – SubjectFull: Programming
        Type: general
      – SubjectFull: Academic Achievement
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Elementary Education
        Type: general
      – SubjectFull: Secondary Education
        Type: general
      – SubjectFull: Postsecondary Education
        Type: general
      – SubjectFull: Time Factors (Learning)
        Type: general
      – SubjectFull: Educational Environment
        Type: general
      – SubjectFull: Research Design
        Type: general
      – SubjectFull: Programming Languages
        Type: general
      – SubjectFull: Teaching Methods
        Type: general
      – SubjectFull: Influences
        Type: general
    Titles:
      – TitleFull: Do AI Chatbots Improve Students' Learning Performance in Programming Education? Evidence from a Meta-Analysis
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Hongji Deng
      – PersonEntity:
          Name:
            NameFull: Hui Chen
      – PersonEntity:
          Name:
            NameFull: Yan Dong
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 07
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 0735-6331
            – Type: issn-electronic
              Value: 1541-4140
          Numbering:
            – Type: volume
              Value: 64
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: Journal of Educational Computing Research
              Type: main
ResultId 1