Understanding code semantics: a benchmark study of LLMs.
Saved in:
| Title: | Understanding code semantics: a benchmark study of LLMs. |
|---|---|
| Authors: | Laneve, Cosimo1 (AUTHOR) cosimo.laneve@unibo.it, Spanò, Alvise2 (AUTHOR) alvise.spano@unive.it, Ressi, Dalila2 (AUTHOR) dalila.ressi@unive.it, Rossi, Sabina2 (AUTHOR) sabina.rossi@unive.it, Bugliesi, Michele2 (AUTHOR) michele.bugliesi@unive.it |
| Source: | International Journal on Software Tools for Technology Transfer. Jun2026, Vol. 28 Issue 3, p329-343. 15p. |
| Subjects: | Programming language semantics, Program transformation, Benchmark problems (Computer science), Python programming language, Language models |
| Abstract: | We present an empirical study on the ability of Large Language Models (LLMs) to understand code by detecting semantically equivalent and inequivalent programs, that is, whether they compute the same result given the same input or not. To probe this, we deliberately perturb the program text by introducing semantics-preserving code transformations, namely copy propagation and constant folding. Using a benchmark of 11 Python functions with both equivalent and non-equivalent variants, we evaluate seven state-of-the-art LLMs (including ChatGPT, Claude, Gemini, and Deep-Seek) under zero-shot prompting, with and without minimal context. Despite strong performance in code generation tasks, the models often fail in this deeper reasoning challenge, misclassifying 41% of equivalent cases without context and 29% with context. Although prompting can improve performance, it does not address the underlying limitations of the models. We argue that improving LLMs themselves, through targeted fine-tuning, contrastive learning on equivalent and nonequivalent implementations, or training on transformation-invariant code, will be necessary for robust semantic understanding. Meanwhile, practitioners can achieve better results by selecting stronger models, carefully engineering prom-pts, or writing code with tools that normalize low-level differences before inference. [ABSTRACT FROM AUTHOR] |
| Copyright of International Journal on Software Tools for Technology Transfer is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 194640826 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Understanding code semantics: a benchmark study of LLMs. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Laneve%2C+Cosimo%22">Laneve, Cosimo</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> cosimo.laneve@unibo.it</i><br /><searchLink fieldCode="AR" term="%22Spanò%2C+Alvise%22">Spanò, Alvise</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> alvise.spano@unive.it</i><br /><searchLink fieldCode="AR" term="%22Ressi%2C+Dalila%22">Ressi, Dalila</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> dalila.ressi@unive.it</i><br /><searchLink fieldCode="AR" term="%22Rossi%2C+Sabina%22">Rossi, Sabina</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> sabina.rossi@unive.it</i><br /><searchLink fieldCode="AR" term="%22Bugliesi%2C+Michele%22">Bugliesi, Michele</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> michele.bugliesi@unive.it</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22International+Journal+on+Software+Tools+for+Technology+Transfer%22">International Journal on Software Tools for Technology Transfer</searchLink>. Jun2026, Vol. 28 Issue 3, p329-343. 15p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Programming+language+semantics%22">Programming language semantics</searchLink><br /><searchLink fieldCode="DE" term="%22Program+transformation%22">Program transformation</searchLink><br /><searchLink fieldCode="DE" term="%22Benchmark+problems+%28Computer+science%29%22">Benchmark problems (Computer science)</searchLink><br /><searchLink fieldCode="DE" term="%22Python+programming+language%22">Python programming language</searchLink><br /><searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: We present an empirical study on the ability of Large Language Models (LLMs) to understand code by detecting semantically equivalent and inequivalent programs, that is, whether they compute the same result given the same input or not. To probe this, we deliberately perturb the program text by introducing semantics-preserving code transformations, namely copy propagation and constant folding. Using a benchmark of 11 Python functions with both equivalent and non-equivalent variants, we evaluate seven state-of-the-art LLMs (including ChatGPT, Claude, Gemini, and Deep-Seek) under zero-shot prompting, with and without minimal context. Despite strong performance in code generation tasks, the models often fail in this deeper reasoning challenge, misclassifying 41% of equivalent cases without context and 29% with context. Although prompting can improve performance, it does not address the underlying limitations of the models. We argue that improving LLMs themselves, through targeted fine-tuning, contrastive learning on equivalent and nonequivalent implementations, or training on transformation-invariant code, will be necessary for robust semantic understanding. Meanwhile, practitioners can achieve better results by selecting stronger models, carefully engineering prom-pts, or writing code with tools that normalize low-level differences before inference. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of International Journal on Software Tools for Technology Transfer is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=194640826 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1007/s10009-026-00842-4 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 15 StartPage: 329 Subjects: – SubjectFull: Programming language semantics Type: general – SubjectFull: Program transformation Type: general – SubjectFull: Benchmark problems (Computer science) Type: general – SubjectFull: Python programming language Type: general – SubjectFull: Language models Type: general Titles: – TitleFull: Understanding code semantics: a benchmark study of LLMs. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Laneve, Cosimo – PersonEntity: Name: NameFull: Spanò, Alvise – PersonEntity: Name: NameFull: Ressi, Dalila – PersonEntity: Name: NameFull: Rossi, Sabina – PersonEntity: Name: NameFull: Bugliesi, Michele IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 06 Text: Jun2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 14332779 Numbering: – Type: volume Value: 28 – Type: issue Value: 3 Titles: – TitleFull: International Journal on Software Tools for Technology Transfer Type: main |
| ResultId | 1 |
