Smartformer: An intelligent transformer compression framework for time-series modeling.
Saved in:
| Title: | Smartformer: An intelligent transformer compression framework for time-series modeling. |
|---|---|
| Authors: | Wang, Xiaojian1 (AUTHOR), Wang, Yinan2 (AUTHOR), Yang, Jin1 (AUTHOR), Chen, Ying1 (AUTHOR) yingchen@hit.edu.cn |
| Source: | IISE Transactions. Sep2025, Vol. 57 Issue 9, p1027-1040. 14p. |
| Subjects: | Artificial neural networks, Deep reinforcement learning, Transformer models, Resource-limited settings, Time series analysis |
| Abstract: | Transformer, as one of the cutting-edge deep neural networks (DNNs), has achieved outstanding performance in time-series data analysis. However, this model usually requires large numbers of parameters to fit. Over-parameterization not only brings storage challenges in a resource-limited setting, but also inevitably results in the model over-fitting. Even though literature works introduced several ways to reduce the parameter size of Transformers, none of them addressed this over-parameterized issue by concurrently achieving the following three objectives: preserving the model architecture, maintaining the model performance, and reducing the model complexity (number of parameters). In this study, we propose an intelligent model compression framework, Smartformer, by incorporating reinforcement learning and CP-decomposition techniques to satisfy the aforementioned three objectives. In the experiment, we apply Smartformer and five baseline methods to two existing time-series Transformer models for model compression. The results demonstrate that our proposed Smartformer is the only method that consistently generates the compressed model on various scenarios by satisfying the three objectives. In particular, the Smartformer can mitigate the overfitting issue and thus improve the accuracy of the existing time-series models in all scenarios. [ABSTRACT FROM AUTHOR] |
| Copyright of IISE Transactions is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Transformer, as one of the cutting-edge deep neural networks (DNNs), has achieved outstanding performance in time-series data analysis. However, this model usually requires large numbers of parameters to fit. Over-parameterization not only brings storage challenges in a resource-limited setting, but also inevitably results in the model over-fitting. Even though literature works introduced several ways to reduce the parameter size of Transformers, none of them addressed this over-parameterized issue by concurrently achieving the following three objectives: preserving the model architecture, maintaining the model performance, and reducing the model complexity (number of parameters). In this study, we propose an intelligent model compression framework, Smartformer, by incorporating reinforcement learning and CP-decomposition techniques to satisfy the aforementioned three objectives. In the experiment, we apply Smartformer and five baseline methods to two existing time-series Transformer models for model compression. The results demonstrate that our proposed Smartformer is the only method that consistently generates the compressed model on various scenarios by satisfying the three objectives. In particular, the Smartformer can mitigate the overfitting issue and thus improve the accuracy of the existing time-series models in all scenarios. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 24725854 |
| DOI: | 10.1080/24725854.2024.2376645 |
