Repairing order-dependent flaky tests via test generation.
Saved in:
| Title: | Repairing order-dependent flaky tests via test generation. |
|---|---|
| Authors: | Li, Chengpeng1 chengpengli@utexas.edu, Zhu, Chenguang1 cgzhu@utexas.edu, Wang, Wenxi1 wenxiw@utexas.edu, Shi, August1 august@utexas.edu |
| Source: | ICSE: International Conference on Software Engineering. 2022, p1881-1892. 12p. |
| Subjects: | Source code, Test generators, Computer software, Automatic test equipment, Regression analysis |
| Abstract: | Flaky tests are tests that pass or fail nondeterministically on the same version of code. These tests can mislead developers concerning the quality of their code changes during regression testing. A common kind of flaky tests are order-dependent tests, whose pass/fail outcomes depend on the test order in which they are run. Such tests have different outcomes because other tests running before them pollute shared state. Prior work has proposed repairing order-dependent tests by searching for existing tests, known as "cleaners", that reset the shared state, allowing the order-dependent test to pass when run after a polluted shared state. The code within a cleaner represents a patch to repair the order-dependent test. However, this technique requires cleaners to already exist in the test suite. We propose ODRepair, an automated technique to repair order-dependent tests even without existing cleaners. The idea is to first determine the exact polluted shared state that results in the order-dependent test to fail and then generate code that can modify and reset the shared state so that the order-dependent test can pass. We focus on shared state through internal heap memory, in particular shared state reachable from static fields. Once we know which static field leads to the pollution, we search for reset-methods in the codebase that can potentially access and modify state reachable from that static field. We then apply an automatic test-generation tool to generate method-call sequences, targeting these reset-methods. Our evaluation on 327 order-dependent tests from a publicly available dataset shows that ODRepair automatically identifies the polluted static field for 181 tests, and it can generate patches for 141 of these tests. Compared against state-of-the-art iFixFlakies, ODRepair can generate patches for 24 tests that iFixFlakies cannot. [ABSTRACT FROM AUTHOR] |
| Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Text: Availability: 1 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 185195824 AccessLevel: 6 PubType: Conference PubTypeId: conference PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Repairing order-dependent flaky tests via test generation. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Li%2C+Chengpeng%22">Li, Chengpeng</searchLink><relatesTo>1</relatesTo><i> chengpengli@utexas.edu</i><br /><searchLink fieldCode="AR" term="%22Zhu%2C+Chenguang%22">Zhu, Chenguang</searchLink><relatesTo>1</relatesTo><i> cgzhu@utexas.edu</i><br /><searchLink fieldCode="AR" term="%22Wang%2C+Wenxi%22">Wang, Wenxi</searchLink><relatesTo>1</relatesTo><i> wenxiw@utexas.edu</i><br /><searchLink fieldCode="AR" term="%22Shi%2C+August%22">Shi, August</searchLink><relatesTo>1</relatesTo><i> august@utexas.edu</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22ICSE%3A+International+Conference+on+Software+Engineering%22">ICSE: International Conference on Software Engineering</searchLink>. 2022, p1881-1892. 12p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Source+code%22">Source code</searchLink><br /><searchLink fieldCode="DE" term="%22Test+generators%22">Test generators</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+software%22">Computer software</searchLink><br /><searchLink fieldCode="DE" term="%22Automatic+test+equipment%22">Automatic test equipment</searchLink><br /><searchLink fieldCode="DE" term="%22Regression+analysis%22">Regression analysis</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Flaky tests are tests that pass or fail nondeterministically on the same version of code. These tests can mislead developers concerning the quality of their code changes during regression testing. A common kind of flaky tests are order-dependent tests, whose pass/fail outcomes depend on the test order in which they are run. Such tests have different outcomes because other tests running before them pollute shared state. Prior work has proposed repairing order-dependent tests by searching for existing tests, known as "cleaners", that reset the shared state, allowing the order-dependent test to pass when run after a polluted shared state. The code within a cleaner represents a patch to repair the order-dependent test. However, this technique requires cleaners to already exist in the test suite. We propose ODRepair, an automated technique to repair order-dependent tests even without existing cleaners. The idea is to first determine the exact polluted shared state that results in the order-dependent test to fail and then generate code that can modify and reset the shared state so that the order-dependent test can pass. We focus on shared state through internal heap memory, in particular shared state reachable from static fields. Once we know which static field leads to the pollution, we search for reset-methods in the codebase that can potentially access and modify state reachable from that static field. We then apply an automatic test-generation tool to generate method-call sequences, targeting these reset-methods. Our evaluation on 327 order-dependent tests from a publicly available dataset shows that ODRepair automatically identifies the polluted static field for 181 tests, and it can generate patches for 141 of these tests. Compared against state-of-the-art iFixFlakies, ODRepair can generate patches for 24 tests that iFixFlakies cannot. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of ICSE: International Conference on Software Engineering is the property of Association for Computing Machinery and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=185195824 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1145/3510003.3510173 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 12 StartPage: 1881 Subjects: – SubjectFull: Source code Type: general – SubjectFull: Test generators Type: general – SubjectFull: Computer software Type: general – SubjectFull: Automatic test equipment Type: general – SubjectFull: Regression analysis Type: general Titles: – TitleFull: Repairing order-dependent flaky tests via test generation. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Li, Chengpeng – PersonEntity: Name: NameFull: Zhu, Chenguang – PersonEntity: Name: NameFull: Wang, Wenxi – PersonEntity: Name: NameFull: Shi, August IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Text: 2022 Type: published Y: 2022 Titles: – TitleFull: ICSE: International Conference on Software Engineering Type: main |
| ResultId | 1 |
