Do AI Chatbots Improve Students' Learning Performance in Programming Education? Evidence from a Meta-Analysis
Saved in:
| Title: | Do AI Chatbots Improve Students' Learning Performance in Programming Education? Evidence from a Meta-Analysis |
|---|---|
| Language: | English |
| Authors: | Hongji Deng (ORCID |
| Source: | Journal of Educational Computing Research. 2026 64(5):1323-1359. |
| Availability: | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
| Peer Reviewed: | Y |
| Page Count: | 37 |
| Publication Date: | 2026 |
| Document Type: | Journal Articles Information Analyses |
| Education Level: | Elementary Education Secondary Education Postsecondary Education |
| Descriptors: | Artificial Intelligence, Synchronous Communication, Programming, Academic Achievement, Technology Uses in Education, Elementary Education, Secondary Education, Postsecondary Education, Time Factors (Learning), Educational Environment, Research Design, Programming Languages, Teaching Methods, Influences |
| DOI: | 10.1177/07356331261424211 |
| ISSN: | 0735-6331 1541-4140 |
| Abstract: | AI chatbots have emerged as innovative educational tools and drawn increasing attention from educators and researchers in programming education. Although previous research has highlighted potentials of applying AI chatbots in programming education, there is a lack of empirical evidence to understand the overall effects of using AI chatbots in programming learning as well as the critical factors that influence the effects. To fill this gap, this study conducted a meta-analysis of 32 empirical studies published between 2015 and 2025 to examine the overall effect size of applying AI chatbots on programming learning performance and identify significant moderators. The results indicated a small-to-medium effect on posttest performance (g+ = 0.538, 95% CI [0.202, 0.873], p < 0.01) and a medium-to-large effect on practice performance (g+ = 0.650, 95% CI [0.330, 0.970], p < 0.001), based on robust variance estimation models. Moderator analyses revealed that research design and AI chatbot-to-student ratio significantly influenced posttest performance. Specifically, true experimental designs demonstrated significantly larger effects than quasi-experimental designs, and a 1:1 chatbot-student ratio was substantially more effective than a 1:N ratio. These findings underscore the potential of AI chatbots in programming education and offer practical insights for optimizing their integration into instructional design. |
| Abstractor: | As Provided |
| Notes: | https://osf.io/gnk6d |
| Entry Date: | 2026 |
| Accession Number: | EJ1506771 |
| Database: | ERIC |
| Abstract: | AI chatbots have emerged as innovative educational tools and drawn increasing attention from educators and researchers in programming education. Although previous research has highlighted potentials of applying AI chatbots in programming education, there is a lack of empirical evidence to understand the overall effects of using AI chatbots in programming learning as well as the critical factors that influence the effects. To fill this gap, this study conducted a meta-analysis of 32 empirical studies published between 2015 and 2025 to examine the overall effect size of applying AI chatbots on programming learning performance and identify significant moderators. The results indicated a small-to-medium effect on posttest performance (g+ = 0.538, 95% CI [0.202, 0.873], p < 0.01) and a medium-to-large effect on practice performance (g+ = 0.650, 95% CI [0.330, 0.970], p < 0.001), based on robust variance estimation models. Moderator analyses revealed that research design and AI chatbot-to-student ratio significantly influenced posttest performance. Specifically, true experimental designs demonstrated significantly larger effects than quasi-experimental designs, and a 1:1 chatbot-student ratio was substantially more effective than a 1:N ratio. These findings underscore the potential of AI chatbots in programming education and offer practical insights for optimizing their integration into instructional design. |
|---|---|
| ISSN: | 0735-6331 1541-4140 |
| DOI: | 10.1177/07356331261424211 |
