View in EDS

Equidistribution-based training of univariate free knot splines and ReLU neural networks: revised.

Saved in:
Bibliographic Details
Title: Equidistribution-based training of univariate free knot splines and ReLU neural networks: revised.
Authors: Appella, Simone1 (AUTHOR), Arridge, Simon2 (AUTHOR), Budd, Chris1 (AUTHOR), Deveney, Teo3 (AUTHOR), Du, Yan2 (AUTHOR), Kreusser, Lisa Maria1 (AUTHOR)
Source: IMA Journal of Applied Mathematics. Apr2026, Vol. 91 Issue 2, p119-150. 32p.
Subjects: Piecewise linear approximation, Spline theory, Feedforward neural networks, Machine learning, Mathematical regularization, Artificial neural networks, Iterative methods (Mathematics), Interpolation
Abstract: We consider the problem of improving the accuracy, convergence and conditioning of univariate nonlinear function approximations using (mainly) shallow neural networks (NN) with a rectified linear unit (ReLU) activation function. The standard |$L_{2}$| -based approximation problem is ill conditioned and the behaviour of the optimization algorithms used in training these networks degrades rapidly as the width of the network increases. This can lead to significantly poorer approximation in practice than we would expect from the theoretical expressivity of the ReLU NN architecture. Univariate shallow ReLU NNs and traditional approximation methods, such as univariate free knot splines (FKS) span the same function space, and thus have the same theoretical expressivity. However, the FKS representation, both remains well conditioned as the number of knots increases, and can be highly accurate if the knots are correctly placed. We leverage the theory of optimal piecewise linear interpolants to improve the training procedure for both an FKS and a ReLU NN. For the FKS we propose a novel two-level training procedure. First solving the nonlinear problem of finding the optimal knot locations of the interpolating FKS using an equidistribution approach. Then solving the nearly linear, well-conditioned, problem of finding the optimal weights and knots of the FKS. The training of the FKS gives insights into how we can train a ReLU NN effectively to give an equally accurate approximation. To do this we combine the training of the ReLU NN with an equidistribution based loss to find the breakpoints of the ReLU functions, this is then combined with preconditioning the ReLU NN approximation (to take an FKS form) to find the scalings of the ReLU functions. This procedure leads to a fast, well-conditioned and reliable method of finding an accurate shallow ReLU NN approximation to a univariate target function. This method avoids spectral bias and is highly effective for a wide variety of functions. We test this method on a series of regular, singular and rapidly varying target functions and obtain good results, realizing the expressivity of the shallow ReLU network in all cases. We conclude that in the shallow case to gain full expressivity for the ReLU NN we must both find the optimal breakpoints (by equidistribution) and precondition the problem of finding the optimal coefficients. We then extend our results to more general activation functions, and to deeper networks. [ABSTRACT FROM AUTHOR]
Copyright of IMA Journal of Applied Mathematics is the property of Oxford University Press / USA and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
FullText Text:
  Availability: 0
Header DbId: egs
DbLabel: Engineering Source
An: 194756319
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Equidistribution-based training of univariate free knot splines and ReLU neural networks: revised.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Appella%2C+Simone%22">Appella, Simone</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Arridge%2C+Simon%22">Arridge, Simon</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Budd%2C+Chris%22">Budd, Chris</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Deveney%2C+Teo%22">Deveney, Teo</searchLink><relatesTo>3</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Du%2C+Yan%22">Du, Yan</searchLink><relatesTo>2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Kreusser%2C+Lisa+Maria%22">Kreusser, Lisa Maria</searchLink><relatesTo>1</relatesTo> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22IMA+Journal+of+Applied+Mathematics%22">IMA Journal of Applied Mathematics</searchLink>. Apr2026, Vol. 91 Issue 2, p119-150. 32p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Piecewise+linear+approximation%22">Piecewise linear approximation</searchLink><br /><searchLink fieldCode="DE" term="%22Spline+theory%22">Spline theory</searchLink><br /><searchLink fieldCode="DE" term="%22Feedforward+neural+networks%22">Feedforward neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Machine+learning%22">Machine learning</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematical+regularization%22">Mathematical regularization</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+neural+networks%22">Artificial neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Iterative+methods+%28Mathematics%29%22">Iterative methods (Mathematics)</searchLink><br /><searchLink fieldCode="DE" term="%22Interpolation%22">Interpolation</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: We consider the problem of improving the accuracy, convergence and conditioning of univariate nonlinear function approximations using (mainly) shallow neural networks (NN) with a rectified linear unit (ReLU) activation function. The standard |$L_{2}$| -based approximation problem is ill conditioned and the behaviour of the optimization algorithms used in training these networks degrades rapidly as the width of the network increases. This can lead to significantly poorer approximation in practice than we would expect from the theoretical expressivity of the ReLU NN architecture. Univariate shallow ReLU NNs and traditional approximation methods, such as univariate free knot splines (FKS) span the same function space, and thus have the same theoretical expressivity. However, the FKS representation, both remains well conditioned as the number of knots increases, and can be highly accurate if the knots are correctly placed. We leverage the theory of optimal piecewise linear interpolants to improve the training procedure for both an FKS and a ReLU NN. For the FKS we propose a novel two-level training procedure. First solving the nonlinear problem of finding the optimal knot locations of the interpolating FKS using an equidistribution approach. Then solving the nearly linear, well-conditioned, problem of finding the optimal weights and knots of the FKS. The training of the FKS gives insights into how we can train a ReLU NN effectively to give an equally accurate approximation. To do this we combine the training of the ReLU NN with an equidistribution based loss to find the breakpoints of the ReLU functions, this is then combined with preconditioning the ReLU NN approximation (to take an FKS form) to find the scalings of the ReLU functions. This procedure leads to a fast, well-conditioned and reliable method of finding an accurate shallow ReLU NN approximation to a univariate target function. This method avoids spectral bias and is highly effective for a wide variety of functions. We test this method on a series of regular, singular and rapidly varying target functions and obtain good results, realizing the expressivity of the shallow ReLU network in all cases. We conclude that in the shallow case to gain full expressivity for the ReLU NN we must both find the optimal breakpoints (by equidistribution) and precondition the problem of finding the optimal coefficients. We then extend our results to more general activation functions, and to deeper networks. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of IMA Journal of Applied Mathematics is the property of Oxford University Press / USA and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=194756319
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1093/imamat/hxag006
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 32
        StartPage: 119
    Subjects:
      – SubjectFull: Piecewise linear approximation
        Type: general
      – SubjectFull: Spline theory
        Type: general
      – SubjectFull: Feedforward neural networks
        Type: general
      – SubjectFull: Machine learning
        Type: general
      – SubjectFull: Mathematical regularization
        Type: general
      – SubjectFull: Artificial neural networks
        Type: general
      – SubjectFull: Iterative methods (Mathematics)
        Type: general
      – SubjectFull: Interpolation
        Type: general
    Titles:
      – TitleFull: Equidistribution-based training of univariate free knot splines and ReLU neural networks: revised.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Appella, Simone
      – PersonEntity:
          Name:
            NameFull: Arridge, Simon
      – PersonEntity:
          Name:
            NameFull: Budd, Chris
      – PersonEntity:
          Name:
            NameFull: Deveney, Teo
      – PersonEntity:
          Name:
            NameFull: Du, Yan
      – PersonEntity:
          Name:
            NameFull: Kreusser, Lisa Maria
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Text: Apr2026
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-print
              Value: 02724960
          Numbering:
            – Type: volume
              Value: 91
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: IMA Journal of Applied Mathematics
              Type: main
ResultId 1