Function Approximation Guarantees for a Shallow Neural Network Trained by Gradient Flow
Saved in:
| Title: | Function Approximation Guarantees for a Shallow Neural Network Trained by Gradient Flow |
|---|---|
| Authors: | Gentile, Russell |
| Committee Members: | Welper, Gerrit |
| Summary: | This work features an original result linking approximation and optimization theory for deep learning. Several examples from recent literature show that, given the same number of learnable parameters, deep neural networks can approximate richer classes of functions, with better accuracy than classical methods. The bulk of approximation theory results though, are only concerned with the infimum error for all possible parameterizations of a given network size. Their proofs often rely on hand-crafted networks, where the weights and biases are carefully selected. Optimization theory indicates that such models would be difficult or impossible to realize with standard gradient-based training methods. The main result of this thesis proves that, for a single-layer neural network having m parameters, a conservative approximation rate, O(m¼), is achieved with gradient flow training on univariate functions. This is especially noteworthy since we make no assumption of overparameterization, as is typically done with neural tangent kernel (NTK) techniques. The proof relies on an assumption that the H1-norm of the residual error throughout the training process is uniformly bounded. This assumption is justified by numerical experiments which also show that rates beyond 1/4 are achieved in practice, indicating that a sharper theoretical result is most likely possible. Future work will focus on proving that the bounded H1 assumption is not needed and that variations of our main result can also be applied to multi-dimensional cases and deep networks. |
| URL: | https://stars.library.ucf.edu/etd2020/1203 |
| Database: | OpenDissertations |
Be the first to leave a comment!
