View in EDS HTML Full Text PDF Full Text

When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?

Saved in:
Bibliographic Details
Title: When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?
Authors: Chen, Yuxiao1,2 chenyuxiao2021@iscas.ac.cn, Wu, Jingzheng3 jingzheng08@iscas.ac.cn, Ling, Xiang2 lingxiang@iscas.ac.cn, Li, Changjiang4 meet.cjli@gmail.com, Rui, Zhiqing1,2 zhiqing@iscas.ac.cn, Luo, Tianyue2 tianyue@iscas.ac.cn, Wu, Yanjun2 yanjun@iscas.ac.cn
Source: ICSE: International Conference on Software Engineering. 2024, p459-471. 13p.
Subjects: Language models, Automatic programming (Computer science), Open source software, Futures studies, Debugging
Abstract: In recent years, large language models (LLMs) have demonstrated substantial potential in addressing automatic program repair (APR) tasks. However, the current evaluation of these models for APR tasks focuses solely on the limited context of the single function or file where the bug is located, overlooking the valuable information in the repository-level context. This paper investigates the performance of popular LLMs in handling repository-level repair tasks. We introduce RepoBugs, a new benchmark comprising 124 typical repository-level bugs from open-source repositories. Preliminary experiments using GPT3.5 based on the function where the error is located, reveal that the repair rate on RepoBugs is only 22.58%, significantly diverging from the performance of GPT3.5 on function-level bugs in related studies. This underscores the importance of providing repository-level context when addressing bugs at this level. However, the repository-level context offered by the preliminary method often proves redundant and imprecise and easily exceeds the prompt length limit of LLMs. To solve the problem, we propose a simple and universal repository-level context extraction method (RLCE) designed to provide more precise context for repository-level code repair tasks. Evaluations of three mainstream LLMs show that RLCE significantly enhances the ability to repair repository-level bugs. The improvement reaches a maximum of 160% compared to the preliminary method. Additionally, we conduct a comprehensive analysis of the effectiveness and limitations of RLCE, along with the capacity of LLMs to address repository-level bugs, offering valuable insights for future research. [ABSTRACT FROM AUTHOR]
Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
Text:
  Availability: 1
Header DbId: egs
DbLabel: Engineering Source
An: 185196678
AccessLevel: 6
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Chen%2C+Yuxiao%22">Chen, Yuxiao</searchLink><relatesTo>1,2</relatesTo><i> chenyuxiao2021@iscas.ac.cn</i><br /><searchLink fieldCode="AR" term="%22Wu%2C+Jingzheng%22">Wu, Jingzheng</searchLink><relatesTo>3</relatesTo><i> jingzheng08@iscas.ac.cn</i><br /><searchLink fieldCode="AR" term="%22Ling%2C+Xiang%22">Ling, Xiang</searchLink><relatesTo>2</relatesTo><i> lingxiang@iscas.ac.cn</i><br /><searchLink fieldCode="AR" term="%22Li%2C+Changjiang%22">Li, Changjiang</searchLink><relatesTo>4</relatesTo><i> meet.cjli@gmail.com</i><br /><searchLink fieldCode="AR" term="%22Rui%2C+Zhiqing%22">Rui, Zhiqing</searchLink><relatesTo>1,2</relatesTo><i> zhiqing@iscas.ac.cn</i><br /><searchLink fieldCode="AR" term="%22Luo%2C+Tianyue%22">Luo, Tianyue</searchLink><relatesTo>2</relatesTo><i> tianyue@iscas.ac.cn</i><br /><searchLink fieldCode="AR" term="%22Wu%2C+Yanjun%22">Wu, Yanjun</searchLink><relatesTo>2</relatesTo><i> yanjun@iscas.ac.cn</i>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22ICSE%3A+International+Conference+on+Software+Engineering%22">ICSE: International Conference on Software Engineering</searchLink>. 2024, p459-471. 13p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22Automatic+programming+%28Computer+science%29%22">Automatic programming (Computer science)</searchLink><br /><searchLink fieldCode="DE" term="%22Open+source+software%22">Open source software</searchLink><br /><searchLink fieldCode="DE" term="%22Futures+studies%22">Futures studies</searchLink><br /><searchLink fieldCode="DE" term="%22Debugging%22">Debugging</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In recent years, large language models (LLMs) have demonstrated substantial potential in addressing automatic program repair (APR) tasks. However, the current evaluation of these models for APR tasks focuses solely on the limited context of the single function or file where the bug is located, overlooking the valuable information in the repository-level context. This paper investigates the performance of popular LLMs in handling repository-level repair tasks. We introduce RepoBugs, a new benchmark comprising 124 typical repository-level bugs from open-source repositories. Preliminary experiments using GPT3.5 based on the function where the error is located, reveal that the repair rate on RepoBugs is only 22.58%, significantly diverging from the performance of GPT3.5 on function-level bugs in related studies. This underscores the importance of providing repository-level context when addressing bugs at this level. However, the repository-level context offered by the preliminary method often proves redundant and imprecise and easily exceeds the prompt length limit of LLMs. To solve the problem, we propose a simple and universal repository-level context extraction method (RLCE) designed to provide more precise context for repository-level code repair tasks. Evaluations of three mainstream LLMs show that RLCE significantly enhances the ability to repair repository-level bugs. The improvement reaches a maximum of 160% compared to the preliminary method. Additionally, we conduct a comprehensive analysis of the effectiveness and limitations of RLCE, along with the capacity of LLMs to address repository-level bugs, offering valuable insights for future research. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=185196678
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1145/3639478.3647633
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 13
        StartPage: 459
    Subjects:
      – SubjectFull: Language models
        Type: general
      – SubjectFull: Automatic programming (Computer science)
        Type: general
      – SubjectFull: Open source software
        Type: general
      – SubjectFull: Futures studies
        Type: general
      – SubjectFull: Debugging
        Type: general
    Titles:
      – TitleFull: When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done?
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Chen, Yuxiao
      – PersonEntity:
          Name:
            NameFull: Wu, Jingzheng
      – PersonEntity:
          Name:
            NameFull: Ling, Xiang
      – PersonEntity:
          Name:
            NameFull: Li, Changjiang
      – PersonEntity:
          Name:
            NameFull: Rui, Zhiqing
      – PersonEntity:
          Name:
            NameFull: Luo, Tianyue
      – PersonEntity:
          Name:
            NameFull: Wu, Yanjun
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 05
              Text: 2024
              Type: published
              Y: 2024
          Titles:
            – TitleFull: ICSE: International Conference on Software Engineering
              Type: main
ResultId 1