Smartformer: An intelligent transformer compression framework for time-series modeling.
Saved in:
| Title: | Smartformer: An intelligent transformer compression framework for time-series modeling. |
|---|---|
| Authors: | Wang, Xiaojian1 (AUTHOR), Wang, Yinan2 (AUTHOR), Yang, Jin1 (AUTHOR), Chen, Ying1 (AUTHOR) yingchen@hit.edu.cn |
| Source: | IISE Transactions. Sep2025, Vol. 57 Issue 9, p1027-1040. 14p. |
| Subjects: | Artificial neural networks, Deep reinforcement learning, Transformer models, Resource-limited settings, Time series analysis |
| Abstract: | Transformer, as one of the cutting-edge deep neural networks (DNNs), has achieved outstanding performance in time-series data analysis. However, this model usually requires large numbers of parameters to fit. Over-parameterization not only brings storage challenges in a resource-limited setting, but also inevitably results in the model over-fitting. Even though literature works introduced several ways to reduce the parameter size of Transformers, none of them addressed this over-parameterized issue by concurrently achieving the following three objectives: preserving the model architecture, maintaining the model performance, and reducing the model complexity (number of parameters). In this study, we propose an intelligent model compression framework, Smartformer, by incorporating reinforcement learning and CP-decomposition techniques to satisfy the aforementioned three objectives. In the experiment, we apply Smartformer and five baseline methods to two existing time-series Transformer models for model compression. The results demonstrate that our proposed Smartformer is the only method that consistently generates the compressed model on various scenarios by satisfying the three objectives. In particular, the Smartformer can mitigate the overfitting issue and thus improve the accuracy of the existing time-series models in all scenarios. [ABSTRACT FROM AUTHOR] |
| Copyright of IISE Transactions is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Text: Availability: 1 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 185908194 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Smartformer: An intelligent transformer compression framework for time-series modeling. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Wang%2C+Xiaojian%22">Wang, Xiaojian</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Wang%2C+Yinan%22">Wang, Yinan</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Yang%2C+Jin%22">Yang, Jin</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Chen%2C+Ying%22">Chen, Ying</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> yingchen@hit.edu.cn</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22IISE+Transactions%22">IISE Transactions</searchLink>. Sep2025, Vol. 57 Issue 9, p1027-1040. 14p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Artificial+neural+networks%22">Artificial neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Deep+reinforcement+learning%22">Deep reinforcement learning</searchLink><br /><searchLink fieldCode="DE" term="%22Transformer+models%22">Transformer models</searchLink><br /><searchLink fieldCode="DE" term="%22Resource-limited+settings%22">Resource-limited settings</searchLink><br /><searchLink fieldCode="DE" term="%22Time+series+analysis%22">Time series analysis</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Transformer, as one of the cutting-edge deep neural networks (DNNs), has achieved outstanding performance in time-series data analysis. However, this model usually requires large numbers of parameters to fit. Over-parameterization not only brings storage challenges in a resource-limited setting, but also inevitably results in the model over-fitting. Even though literature works introduced several ways to reduce the parameter size of Transformers, none of them addressed this over-parameterized issue by concurrently achieving the following three objectives: preserving the model architecture, maintaining the model performance, and reducing the model complexity (number of parameters). In this study, we propose an intelligent model compression framework, Smartformer, by incorporating reinforcement learning and CP-decomposition techniques to satisfy the aforementioned three objectives. In the experiment, we apply Smartformer and five baseline methods to two existing time-series Transformer models for model compression. The results demonstrate that our proposed Smartformer is the only method that consistently generates the compressed model on various scenarios by satisfying the three objectives. In particular, the Smartformer can mitigate the overfitting issue and thus improve the accuracy of the existing time-series models in all scenarios. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of IISE Transactions is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=185908194 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1080/24725854.2024.2376645 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 14 StartPage: 1027 Subjects: – SubjectFull: Artificial neural networks Type: general – SubjectFull: Deep reinforcement learning Type: general – SubjectFull: Transformer models Type: general – SubjectFull: Resource-limited settings Type: general – SubjectFull: Time series analysis Type: general Titles: – TitleFull: Smartformer: An intelligent transformer compression framework for time-series modeling. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Wang, Xiaojian – PersonEntity: Name: NameFull: Wang, Yinan – PersonEntity: Name: NameFull: Yang, Jin – PersonEntity: Name: NameFull: Chen, Ying IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 09 Text: Sep2025 Type: published Y: 2025 Identifiers: – Type: issn-print Value: 24725854 Numbering: – Type: volume Value: 57 – Type: issue Value: 9 Titles: – TitleFull: IISE Transactions Type: main |
| ResultId | 1 |
