On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot.
Saved in:
| Title: | On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot. |
|---|---|
| Authors: | Mastropaolo, Antonio1, Pascarella, Luca1, Guglielmi, Emanuela2, Ciniselli, Matteo1, Scalabrino, Simone2, Oliveto, Rocco2, Bavota, Gabriele1 |
| Source: | ICSE: International Conference on Software Engineering. 2023, p2149-2160. 12p. |
| Subjects: | Github Inc., Software engineering, Deep learning, Recommender systems, Robust statistics |
| Abstract: | Software engineering research has always being concerned with the improvement of code completion approaches, which suggest the next tokens a developer will likely type while coding. The release of GitHub Copilot constitutes a big step forward, also because of its unprecedented ability to automatically generate even entire functions from their natural language description. While the usefulness of Copilot is evident, it is still unclear to what extent it is robust. Specifically, we do not know the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code function. In this paper we present an empirical study in which we aim at understanding whether different but semantically equivalent natural language descriptions result in the same recommended function. A negative answer would pose questions on the robustness of deep learning (DL)-based code generators since it would imply that developers using different wordings to describe the same code would obtain different recommendations. We asked Copilot to automatically generate 892 Java methods starting from their original Javadoc description. Then, we generated different semantically equivalent descriptions for each method both manually and automatically, and we analyzed the extent to which predictions generated by Copilot changed. Our results show that modifying the description results in different code recommendations in ~46% of cases. Also, differences in the semantically equivalent descriptions might impact the correctness of the generated code (±28%). [ABSTRACT FROM AUTHOR] |
| Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Text: Availability: 1 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 185196151 AccessLevel: 6 PubType: Conference PubTypeId: conference PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Mastropaolo%2C+Antonio%22">Mastropaolo, Antonio</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Pascarella%2C+Luca%22">Pascarella, Luca</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Guglielmi%2C+Emanuela%22">Guglielmi, Emanuela</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Ciniselli%2C+Matteo%22">Ciniselli, Matteo</searchLink><relatesTo>1</relatesTo><br /><searchLink fieldCode="AR" term="%22Scalabrino%2C+Simone%22">Scalabrino, Simone</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Oliveto%2C+Rocco%22">Oliveto, Rocco</searchLink><relatesTo>2</relatesTo><br /><searchLink fieldCode="AR" term="%22Bavota%2C+Gabriele%22">Bavota, Gabriele</searchLink><relatesTo>1</relatesTo> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22ICSE%3A+International+Conference+on+Software+Engineering%22">ICSE: International Conference on Software Engineering</searchLink>. 2023, p2149-2160. 12p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Github+Inc%2E%22">Github Inc.</searchLink><br /><searchLink fieldCode="DE" term="%22Software+engineering%22">Software engineering</searchLink><br /><searchLink fieldCode="DE" term="%22Deep+learning%22">Deep learning</searchLink><br /><searchLink fieldCode="DE" term="%22Recommender+systems%22">Recommender systems</searchLink><br /><searchLink fieldCode="DE" term="%22Robust+statistics%22">Robust statistics</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Software engineering research has always being concerned with the improvement of code completion approaches, which suggest the next tokens a developer will likely type while coding. The release of GitHub Copilot constitutes a big step forward, also because of its unprecedented ability to automatically generate even entire functions from their natural language description. While the usefulness of Copilot is evident, it is still unclear to what extent it is robust. Specifically, we do not know the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code function. In this paper we present an empirical study in which we aim at understanding whether different but semantically equivalent natural language descriptions result in the same recommended function. A negative answer would pose questions on the robustness of deep learning (DL)-based code generators since it would imply that developers using different wordings to describe the same code would obtain different recommendations. We asked Copilot to automatically generate 892 Java methods starting from their original Javadoc description. Then, we generated different semantically equivalent descriptions for each method both manually and automatically, and we analyzed the extent to which predictions generated by Copilot changed. Our results show that modifying the description results in different code recommendations in ~46% of cases. Also, differences in the semantically equivalent descriptions might impact the correctness of the generated code (±28%). [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=185196151 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1109/ICSE48619.2023.00181 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 12 StartPage: 2149 Subjects: – SubjectFull: Github Inc. Type: general – SubjectFull: Software engineering Type: general – SubjectFull: Deep learning Type: general – SubjectFull: Recommender systems Type: general – SubjectFull: Robust statistics Type: general Titles: – TitleFull: On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Mastropaolo, Antonio – PersonEntity: Name: NameFull: Pascarella, Luca – PersonEntity: Name: NameFull: Guglielmi, Emanuela – PersonEntity: Name: NameFull: Ciniselli, Matteo – PersonEntity: Name: NameFull: Scalabrino, Simone – PersonEntity: Name: NameFull: Oliveto, Rocco – PersonEntity: Name: NameFull: Bavota, Gabriele IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Text: 2023 Type: published Y: 2023 Titles: – TitleFull: ICSE: International Conference on Software Engineering Type: main |
| ResultId | 1 |
