On the Robustness of Transformer‐Based Models to Different Linguistic Perturbations: A Case of Study in Irony Detection.
Saved in:
| Title: | On the Robustness of Transformer‐Based Models to Different Linguistic Perturbations: A Case of Study in Irony Detection. |
|---|---|
| Authors: | Ortega‐Bueno, Reynier1 (AUTHOR) rortega@prhlt.upv.es, Fersini, Elisabetta2 (AUTHOR), Rosso, Paolo1,3 (AUTHOR) |
| Source: | Expert Systems. Jun2025, Vol. 42 Issue 6, p1-24. 24p. |
| Subjects: | Language models, Linguistic models, Paraphrase, Data curation, Prediction models |
| Abstract: | This study investigates the robustness of Transformer models in irony detection addressing various textual perturbations, revealing potential biases in training data concerning ironic and non‐ironic classes. The perturbations involve three distinct approaches, each progressively increasing in complexity. The first approach is word masking, which employs wild‐card characters or utilises BERT‐specific masking through the mask token provided by BERT models. The second approach is word substitution, replacing the bias word with a contextually appropriate alternative. Lastly, paraphrasing generates a new phrase while preserving the original semantic meaning. We leverage Large Language Models (GPT 3.5 Turbo) and human inspection to ensure linguistic correctness and contextual coherence for word substitutions and paraphrasing. The results indicate that models are susceptible to these perturbations, and paraphrasing and word substitution demonstrate the most significant impact on model predictions. The irony class appears to be particularly challenging for models when subjected to these perturbations. The SHAP and LIME methods are used to correlate variations in attribution scores with prediction errors. A notable difference in the Total Variation of attribution scores is observed between original examples and cases involving bias word substitution or masking. Among the corpora used, TwSemEval2018 emerges as the most challenging. Regarding model performance, Transformer‐based models such as RoBERTa and BERTweet demonstrate superior overall performance addressing these perturbations. This research contributes to understanding the robustness and limitations of irony detection models, highlighting areas for improvement in model design and training data curation. [ABSTRACT FROM AUTHOR] |
| Copyright of Expert Systems is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Text: Availability: 1 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 185122817 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: On the Robustness of Transformer‐Based Models to Different Linguistic Perturbations: A Case of Study in Irony Detection. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Ortega‐Bueno%2C+Reynier%22">Ortega‐Bueno, Reynier</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> rortega@prhlt.upv.es</i><br /><searchLink fieldCode="AR" term="%22Fersini%2C+Elisabetta%22">Fersini, Elisabetta</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Rosso%2C+Paolo%22">Rosso, Paolo</searchLink><relatesTo>1,3</relatesTo> (AUTHOR) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Expert+Systems%22">Expert Systems</searchLink>. Jun2025, Vol. 42 Issue 6, p1-24. 24p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22Linguistic+models%22">Linguistic models</searchLink><br /><searchLink fieldCode="DE" term="%22Paraphrase%22">Paraphrase</searchLink><br /><searchLink fieldCode="DE" term="%22Data+curation%22">Data curation</searchLink><br /><searchLink fieldCode="DE" term="%22Prediction+models%22">Prediction models</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: This study investigates the robustness of Transformer models in irony detection addressing various textual perturbations, revealing potential biases in training data concerning ironic and non‐ironic classes. The perturbations involve three distinct approaches, each progressively increasing in complexity. The first approach is word masking, which employs wild‐card characters or utilises BERT‐specific masking through the mask token provided by BERT models. The second approach is word substitution, replacing the bias word with a contextually appropriate alternative. Lastly, paraphrasing generates a new phrase while preserving the original semantic meaning. We leverage Large Language Models (GPT 3.5 Turbo) and human inspection to ensure linguistic correctness and contextual coherence for word substitutions and paraphrasing. The results indicate that models are susceptible to these perturbations, and paraphrasing and word substitution demonstrate the most significant impact on model predictions. The irony class appears to be particularly challenging for models when subjected to these perturbations. The SHAP and LIME methods are used to correlate variations in attribution scores with prediction errors. A notable difference in the Total Variation of attribution scores is observed between original examples and cases involving bias word substitution or masking. Among the corpora used, TwSemEval2018 emerges as the most challenging. Regarding model performance, Transformer‐based models such as RoBERTa and BERTweet demonstrate superior overall performance addressing these perturbations. This research contributes to understanding the robustness and limitations of irony detection models, highlighting areas for improvement in model design and training data curation. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Expert Systems is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=185122817 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1111/exsy.70062 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 24 StartPage: 1 Subjects: – SubjectFull: Language models Type: general – SubjectFull: Linguistic models Type: general – SubjectFull: Paraphrase Type: general – SubjectFull: Data curation Type: general – SubjectFull: Prediction models Type: general Titles: – TitleFull: On the Robustness of Transformer‐Based Models to Different Linguistic Perturbations: A Case of Study in Irony Detection. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Ortega‐Bueno, Reynier – PersonEntity: Name: NameFull: Fersini, Elisabetta – PersonEntity: Name: NameFull: Rosso, Paolo IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 06 Text: Jun2025 Type: published Y: 2025 Identifiers: – Type: issn-print Value: 02664720 Numbering: – Type: volume Value: 42 – Type: issue Value: 6 Titles: – TitleFull: Expert Systems Type: main |
| ResultId | 1 |
