Reasoning Runtime Behavior of a Program with LLM: How Far Are We?
Saved in:
| Title: | Reasoning Runtime Behavior of a Program with LLM: How Far Are We? |
|---|---|
| Authors: | Chen, Junkai1 junkaichen@zju.edu.cn, Pan, Zhiyuan2 zy_pan@zju.edu.cn, Hu, Xing2 xinghu@zju.edu.cn, Li, Zhenhao3 zhenhao.li@ieee.org, Li, Ge4 lige@pku.edu.cn, Xia, Xin5 xin.xia@acm.org |
| Source: | ICSE: International Conference on Software Engineering. 2025, p2017-2029. 13p. |
| Subjects: | Computer software execution, Evaluation methodology, Language models, Benchmark problems (Computer science) |
| Abstract: | Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs (i.e., predicting code execution behaviors such as program output and execution path), but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning. To address these problems, in this paper, we propose a framework, namely REval, for evaluating code reasoning abilities and consistency of code LLMs with program execution. We utilize existing code benchmarks and adapt them to new benchmarks within our framework. A large-scale empirical study is conducted and most LLMs show unsatisfactory performance on both Runtime Behavior Reasoning (i.e., an average accuracy of 44.4%) and Incremental Consistency Evaluation (i.e., an average IC score of 10.3). Evaluation results of current code LLMs reflect the urgent need for the community to strengthen the code reasoning capability of code LLMs. Our code, data and REval leaderboard are available at https://r-eval.github.io. [ABSTRACT FROM AUTHOR] |
| Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Links: – Type: pdflink Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 194043507 AccessLevel: 6 PubType: Conference PubTypeId: conference PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Reasoning Runtime Behavior of a Program with LLM: How Far Are We? – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Chen%2C+Junkai%22">Chen, Junkai</searchLink><relatesTo>1</relatesTo><i> junkaichen@zju.edu.cn</i><br /><searchLink fieldCode="AR" term="%22Pan%2C+Zhiyuan%22">Pan, Zhiyuan</searchLink><relatesTo>2</relatesTo><i> zy_pan@zju.edu.cn</i><br /><searchLink fieldCode="AR" term="%22Hu%2C+Xing%22">Hu, Xing</searchLink><relatesTo>2</relatesTo><i> xinghu@zju.edu.cn</i><br /><searchLink fieldCode="AR" term="%22Li%2C+Zhenhao%22">Li, Zhenhao</searchLink><relatesTo>3</relatesTo><i> zhenhao.li@ieee.org</i><br /><searchLink fieldCode="AR" term="%22Li%2C+Ge%22">Li, Ge</searchLink><relatesTo>4</relatesTo><i> lige@pku.edu.cn</i><br /><searchLink fieldCode="AR" term="%22Xia%2C+Xin%22">Xia, Xin</searchLink><relatesTo>5</relatesTo><i> xin.xia@acm.org</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22ICSE%3A+International+Conference+on+Software+Engineering%22">ICSE: International Conference on Software Engineering</searchLink>. 2025, p2017-2029. 13p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Computer+software+execution%22">Computer software execution</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluation+methodology%22">Evaluation methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22Benchmark+problems+%28Computer+science%29%22">Benchmark problems (Computer science)</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs (i.e., predicting code execution behaviors such as program output and execution path), but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning. To address these problems, in this paper, we propose a framework, namely REval, for evaluating code reasoning abilities and consistency of code LLMs with program execution. We utilize existing code benchmarks and adapt them to new benchmarks within our framework. A large-scale empirical study is conducted and most LLMs show unsatisfactory performance on both Runtime Behavior Reasoning (i.e., an average accuracy of 44.4%) and Incremental Consistency Evaluation (i.e., an average IC score of 10.3). Evaluation results of current code LLMs reflect the urgent need for the community to strengthen the code reasoning capability of code LLMs. Our code, data and REval leaderboard are available at https://r-eval.github.io. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=194043507 |
| RecordInfo | BibRecord: BibEntity: Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 13 StartPage: 2017 Subjects: – SubjectFull: Computer software execution Type: general – SubjectFull: Evaluation methodology Type: general – SubjectFull: Language models Type: general – SubjectFull: Benchmark problems (Computer science) Type: general Titles: – TitleFull: Reasoning Runtime Behavior of a Program with LLM: How Far Are We? Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Chen, Junkai – PersonEntity: Name: NameFull: Pan, Zhiyuan – PersonEntity: Name: NameFull: Hu, Xing – PersonEntity: Name: NameFull: Li, Zhenhao – PersonEntity: Name: NameFull: Li, Ge – PersonEntity: Name: NameFull: Xia, Xin IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Text: 2025 Type: published Y: 2025 Titles: – TitleFull: ICSE: International Conference on Software Engineering Type: main |
| ResultId | 1 |
