Function Approximation Guarantees for a Shallow Neural Network Trained by Gradient Flow
Saved in:
| Title: | Function Approximation Guarantees for a Shallow Neural Network Trained by Gradient Flow |
|---|---|
| Authors: | Gentile, Russell |
| Committee Members: | Welper, Gerrit |
| Summary: | This work features an original result linking approximation and optimization theory for deep learning. Several examples from recent literature show that, given the same number of learnable parameters, deep neural networks can approximate richer classes of functions, with better accuracy than classical methods. The bulk of approximation theory results though, are only concerned with the infimum error for all possible parameterizations of a given network size. Their proofs often rely on hand-crafted networks, where the weights and biases are carefully selected. Optimization theory indicates that such models would be difficult or impossible to realize with standard gradient-based training methods. The main result of this thesis proves that, for a single-layer neural network having m parameters, a conservative approximation rate, O(m¼), is achieved with gradient flow training on univariate functions. This is especially noteworthy since we make no assumption of overparameterization, as is typically done with neural tangent kernel (NTK) techniques. The proof relies on an assumption that the H1-norm of the residual error throughout the training process is uniformly bounded. This assumption is justified by numerical experiments which also show that rates beyond 1/4 are achieved in practice, indicating that a sharper theoretical result is most likely possible. Future work will focus on proving that the bounded H1 assumption is not needed and that variations of our main result can also be applied to multi-dimensional cases and deep networks. |
| URL: | https://stars.library.ucf.edu/etd2020/1203 |
| Database: | OpenDissertations |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: ddu DbLabel: OpenDissertations An: ddu.oai.stars.library.ucf.edu.etd2020.2202 AccessLevel: 6 PubType: Dissertation/ Thesis PubTypeId: dissertation PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Function Approximation Guarantees for a Shallow Neural Network Trained by Gradient Flow – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Gentile%2C+Russell%22">Gentile, Russell</searchLink> – Name: Author Label: Committee Members Group: Au Data: <searchLink fieldCode="CO" term="%22Welper%2C+Gerrit%22">Welper, Gerrit</searchLink> – Name: Abstract Label: Summary Group: Ab Data: This work features an original result linking approximation and optimization theory for deep learning. Several examples from recent literature show that, given the same number of learnable parameters, deep neural networks can approximate richer classes of functions, with better accuracy than classical methods. The bulk of approximation theory results though, are only concerned with the infimum error for all possible parameterizations of a given network size. Their proofs often rely on hand-crafted networks, where the weights and biases are carefully selected. Optimization theory indicates that such models would be difficult or impossible to realize with standard gradient-based training methods. The main result of this thesis proves that, for a single-layer neural network having m parameters, a conservative approximation rate, O(m¼), is achieved with gradient flow training on univariate functions. This is especially noteworthy since we make no assumption of overparameterization, as is typically done with neural tangent kernel (NTK) techniques. The proof relies on an assumption that the H1-norm of the residual error throughout the training process is uniformly bounded. This assumption is justified by numerical experiments which also show that rates beyond 1/4 are achieved in practice, indicating that a sharper theoretical result is most likely possible. Future work will focus on proving that the bounded H1 assumption is not needed and that variations of our main result can also be applied to multi-dimensional cases and deep networks. – Name: URL Label: URL Group: URL Data: <link linkTarget="URL" linkTerm="https://stars.library.ucf.edu/etd2020/1203" linkWindow="_blank">https://stars.library.ucf.edu/etd2020/1203</link> |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ddu&AN=ddu.oai.stars.library.ucf.edu.etd2020.2202 |
| RecordInfo | BibRecord: BibEntity: Languages: – Code: eng Text: English Subjects: – SubjectFull: Gradient flow; Function approximation; Neural networks; Optimization theory; H1-norm Type: general – SubjectFull: Mathematics Type: general – SubjectFull: Approximation theory; Neural networks (Computer science)--Mathematical models; Neural networks (Computer science)--Research; Neural networks (Computer science)--Statistical methods; Flows (Differentiable dynamical systems) Type: general Titles: – TitleFull: Function Approximation Guarantees for a Shallow Neural Network Trained by Gradient Flow Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Gentile, Russell IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2022 |
| ResultId | 1 |
