A Study on Truncated Newton Methods for Linear Classification.
Saved in:
| Title: | A Study on Truncated Newton Methods for Linear Classification. |
|---|---|
| Authors: | Galli, Leonardo1 (AUTHOR) leonardo.galli@unifi.it, Lin, Chih-Jen2 (AUTHOR) cjlin@csie.ntu.edu.tw |
| Source: | IEEE Transactions on Neural Networks & Learning Systems. Jul2022, Vol. 33 Issue 7, p2828-2841. 14p. |
| Subjects: | Newton-Raphson method, Classification |
| Abstract: | Truncated Newton (TN) methods have been a useful technique for large-scale optimization. Instead of obtaining the full Newton direction, a truncated method approximately solves the Newton equation with an inner conjugate gradient (CG) procedure (TNCG for the whole method). These methods have been employed to efficiently solve linear classification problems. However, even in this deeply studied field, various theoretical and numerical aspects were not completely explored. The first contribution of this work is to comprehensively study the global and local convergence when TNCG is applied to linear classification. Because of the lack of twice differentiability under some losses, many past works cannot be applied here. We prove various missing pieces of theory from scratch and clarify many proper references. The second contribution is to study the termination of the CG method. For the first time when TNCG is applied to linear classification, we show that the inner stopping condition strongly affects the convergence speed. We propose using a quadratic stopping criterion to achieve both robustness and efficiency. The third contribution is that of combining the study on inner stopping criteria with that of preconditioning. We discuss how convergence theory is affected by preconditioning and finally propose an effective preconditioned TNCG. [ABSTRACT FROM AUTHOR] |
| Copyright of IEEE Transactions on Neural Networks & Learning Systems is the property of IEEE and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Engineering Source |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: egs DbLabel: Engineering Source An: 157841102 AccessLevel: 6 PubType: Periodical PubTypeId: serialPeriodical PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: A Study on Truncated Newton Methods for Linear Classification. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Galli%2C+Leonardo%22">Galli, Leonardo</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> leonardo.galli@unifi.it</i><br /><searchLink fieldCode="AR" term="%22Lin%2C+Chih-Jen%22">Lin, Chih-Jen</searchLink><relatesTo>2</relatesTo> (AUTHOR)<i> cjlin@csie.ntu.edu.tw</i> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22IEEE+Transactions+on+Neural+Networks+%26+Learning+Systems%22">IEEE Transactions on Neural Networks & Learning Systems</searchLink>. Jul2022, Vol. 33 Issue 7, p2828-2841. 14p. – Name: Subject Label: Subjects Group: Su Data: <searchLink fieldCode="DE" term="%22Newton-Raphson+method%22">Newton-Raphson method</searchLink><br /><searchLink fieldCode="DE" term="%22Classification%22">Classification</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Truncated Newton (TN) methods have been a useful technique for large-scale optimization. Instead of obtaining the full Newton direction, a truncated method approximately solves the Newton equation with an inner conjugate gradient (CG) procedure (TNCG for the whole method). These methods have been employed to efficiently solve linear classification problems. However, even in this deeply studied field, various theoretical and numerical aspects were not completely explored. The first contribution of this work is to comprehensively study the global and local convergence when TNCG is applied to linear classification. Because of the lack of twice differentiability under some losses, many past works cannot be applied here. We prove various missing pieces of theory from scratch and clarify many proper references. The second contribution is to study the termination of the CG method. For the first time when TNCG is applied to linear classification, we show that the inner stopping condition strongly affects the convergence speed. We propose using a quadratic stopping criterion to achieve both robustness and efficiency. The third contribution is that of combining the study on inner stopping criteria with that of preconditioning. We discuss how convergence theory is affected by preconditioning and finally propose an effective preconditioned TNCG. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of IEEE Transactions on Neural Networks & Learning Systems is the property of IEEE and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=157841102 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1109/TNNLS.2020.3045836 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 14 StartPage: 2828 Subjects: – SubjectFull: Newton-Raphson method Type: general – SubjectFull: Classification Type: general Titles: – TitleFull: A Study on Truncated Newton Methods for Linear Classification. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Galli, Leonardo – PersonEntity: Name: NameFull: Lin, Chih-Jen IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 07 Text: Jul2022 Type: published Y: 2022 Identifiers: – Type: issn-print Value: 2162237X Numbering: – Type: volume Value: 33 – Type: issue Value: 7 Titles: – TitleFull: IEEE Transactions on Neural Networks & Learning Systems Type: main |
| ResultId | 1 |
