View in EDS

Equidistribution-based training of univariate free knot splines and ReLU neural networks: revised.

Saved in:

Bibliographic Details
Title:	Equidistribution-based training of univariate free knot splines and ReLU neural networks: revised.
Authors:	Appella, Simone¹ (AUTHOR), Arridge, Simon² (AUTHOR), Budd, Chris¹ (AUTHOR), Deveney, Teo³ (AUTHOR), Du, Yan² (AUTHOR), Kreusser, Lisa Maria¹ (AUTHOR)
Source:	IMA Journal of Applied Mathematics. Apr2026, Vol. 91 Issue 2, p119-150. 32p.
Subjects:	Piecewise linear approximation, Spline theory, Feedforward neural networks, Machine learning, Mathematical regularization, Artificial neural networks, Iterative methods (Mathematics), Interpolation
Abstract:	We consider the problem of improving the accuracy, convergence and conditioning of univariate nonlinear function approximations using (mainly) shallow neural networks (NN) with a rectified linear unit (ReLU) activation function. The standard \|$L_{2}$\| -based approximation problem is ill conditioned and the behaviour of the optimization algorithms used in training these networks degrades rapidly as the width of the network increases. This can lead to significantly poorer approximation in practice than we would expect from the theoretical expressivity of the ReLU NN architecture. Univariate shallow ReLU NNs and traditional approximation methods, such as univariate free knot splines (FKS) span the same function space, and thus have the same theoretical expressivity. However, the FKS representation, both remains well conditioned as the number of knots increases, and can be highly accurate if the knots are correctly placed. We leverage the theory of optimal piecewise linear interpolants to improve the training procedure for both an FKS and a ReLU NN. For the FKS we propose a novel two-level training procedure. First solving the nonlinear problem of finding the optimal knot locations of the interpolating FKS using an equidistribution approach. Then solving the nearly linear, well-conditioned, problem of finding the optimal weights and knots of the FKS. The training of the FKS gives insights into how we can train a ReLU NN effectively to give an equally accurate approximation. To do this we combine the training of the ReLU NN with an equidistribution based loss to find the breakpoints of the ReLU functions, this is then combined with preconditioning the ReLU NN approximation (to take an FKS form) to find the scalings of the ReLU functions. This procedure leads to a fast, well-conditioned and reliable method of finding an accurate shallow ReLU NN approximation to a univariate target function. This method avoids spectral bias and is highly effective for a wide variety of functions. We test this method on a series of regular, singular and rapidly varying target functions and obtain good results, realizing the expressivity of the shallow ReLU network in all cases. We conclude that in the shallow case to gain full expressivity for the ReLU NN we must both find the optimal breakpoints (by equidistribution) and precondition the problem of finding the optimal coefficients. We then extend our results to more general activation functions, and to deeper networks. [ABSTRACT FROM AUTHOR]
	Copyright of IMA Journal of Applied Mathematics is the property of Oxford University Press / USA and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database:	Engineering Source

Description
Abstract:	We consider the problem of improving the accuracy, convergence and conditioning of univariate nonlinear function approximations using (mainly) shallow neural networks (NN) with a rectified linear unit (ReLU) activation function. The standard \|$L_{2}$\| -based approximation problem is ill conditioned and the behaviour of the optimization algorithms used in training these networks degrades rapidly as the width of the network increases. This can lead to significantly poorer approximation in practice than we would expect from the theoretical expressivity of the ReLU NN architecture. Univariate shallow ReLU NNs and traditional approximation methods, such as univariate free knot splines (FKS) span the same function space, and thus have the same theoretical expressivity. However, the FKS representation, both remains well conditioned as the number of knots increases, and can be highly accurate if the knots are correctly placed. We leverage the theory of optimal piecewise linear interpolants to improve the training procedure for both an FKS and a ReLU NN. For the FKS we propose a novel two-level training procedure. First solving the nonlinear problem of finding the optimal knot locations of the interpolating FKS using an equidistribution approach. Then solving the nearly linear, well-conditioned, problem of finding the optimal weights and knots of the FKS. The training of the FKS gives insights into how we can train a ReLU NN effectively to give an equally accurate approximation. To do this we combine the training of the ReLU NN with an equidistribution based loss to find the breakpoints of the ReLU functions, this is then combined with preconditioning the ReLU NN approximation (to take an FKS form) to find the scalings of the ReLU functions. This procedure leads to a fast, well-conditioned and reliable method of finding an accurate shallow ReLU NN approximation to a univariate target function. This method avoids spectral bias and is highly effective for a wide variety of functions. We test this method on a series of regular, singular and rapidly varying target functions and obtain good results, realizing the expressivity of the shallow ReLU network in all cases. We conclude that in the shallow case to gain full expressivity for the ReLU NN we must both find the optimal breakpoints (by equidistribution) and precondition the problem of finding the optimal coefficients. We then extend our results to more general activation functions, and to deeper networks. [ABSTRACT FROM AUTHOR]
ISSN:	02724960
DOI:	10.1093/imamat/hxag006