Speeding up L-BFGS by direct approximation of the inverse Hessian matrix: Speeding up L-BFGS by direct approximation...: A. Sadeghi-Lotfabadi, K. Ghiasi-Shirazi.
Saved in:
| Title: | Speeding up L-BFGS by direct approximation of the inverse Hessian matrix: Speeding up L-BFGS by direct approximation...: A. Sadeghi-Lotfabadi, K. Ghiasi-Shirazi. |
|---|---|
| Authors: | Sadeghi-Lotfabadi, Ashkan1 (AUTHOR) sadeghia@mail.um.ac.ir, Ghiasi-Shirazi, Kamaledin1 (AUTHOR) k.ghiasi@um.ac.ir |
| Source: | Computational Optimization & Applications. May2025, Vol. 91 Issue 1, p283-310. 28p. |
| Subjects: | SIMD (Computer architecture), Computational mathematics, Quasi-Newton methods, Matrix inversion, Linear operators |
| Abstract: | L-BFGS is one of the widely used quasi-Newton methods. Instead of explicitly storing an approximation H of the inverse Hessian, L-BFGS keeps a limited number of vectors that can be used for computing the product of H by the gradient. However, this computation is sequential, each step depending on the outcome of the previous step. To solve this problem, we propose the Direct L-BFGS (DirL-BFGS) method that, seeing H as a linear operator, directly stores a low-rank plus diagonal (LRPD) representation of H. Employing the LRPD representation enables us to leverage the benefits of vector processing, leading to accelerating and parallelizing the calculations in the form of single instruction, multiple data. We evaluate our proposed method on different quadratic optimization problems and several regression and classification tasks with neural networks. Numerical results show that DirL-BFGS is faster overall than L-BFGS. [ABSTRACT FROM AUTHOR] |
| Copyright of Computational Optimization & Applications is the property of Springer Nature and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | L-BFGS is one of the widely used quasi-Newton methods. Instead of explicitly storing an approximation H of the inverse Hessian, L-BFGS keeps a limited number of vectors that can be used for computing the product of H by the gradient. However, this computation is sequential, each step depending on the outcome of the previous step. To solve this problem, we propose the Direct L-BFGS (DirL-BFGS) method that, seeing H as a linear operator, directly stores a low-rank plus diagonal (LRPD) representation of H. Employing the LRPD representation enables us to leverage the benefits of vector processing, leading to accelerating and parallelizing the calculations in the form of single instruction, multiple data. We evaluate our proposed method on different quadratic optimization problems and several regression and classification tasks with neural networks. Numerical results show that DirL-BFGS is faster overall than L-BFGS. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 09266003 |
| DOI: | 10.1007/s10589-025-00665-0 |
