View in EDS

Stabilizing machine learning prediction of dynamics: Novel noise-inspired regularization tested with reservoir computing.

Saved in:
Bibliographic Details
Title: Stabilizing machine learning prediction of dynamics: Novel noise-inspired regularization tested with reservoir computing.
Authors: Wikner, Alexander1 (AUTHOR) awikner1@umd.edu, Harvey, Joseph1,2 (AUTHOR), Girvan, Michelle1 (AUTHOR), Hunt, Brian R.3 (AUTHOR), Pomerance, Andrew4 (AUTHOR), Antonsen, Thomas1,5 (AUTHOR), Ott, Edward1,5 (AUTHOR)
Source: Neural Networks. Feb2024, Vol. 170, p94-110. 17p.
Subjects: Machine learning, Recurrent neural networks, Biologically inspired computing, Long-Term Evolution (Telecommunications)
Abstract: Recent work has shown that machine learning (ML) models can skillfully forecast the dynamics of unknown chaotic systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics ("climate") can be produced by employing a feedback loop, whereby the model is trained to predict forward only one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this feedback can result in artificially rapid error growth ("instability"). One established mitigating technique is to add noise to the ML model training input. Based on this technique, we formulate a new penalty term in the loss function for ML models with memory of past inputs that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. We refer to this penalty and the resulting regularization as Linearized Multi-Noise Training (LMNT). We systematically examine the effect of LMNT, input noise, and other established regularization techniques in a case study using reservoir computing, a machine learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto–Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while the short-term forecasts are substantially more accurate than those trained with other regularization techniques. Finally, we show the deterministic aspect of our LMNT regularization facilitates fast reservoir computer regularization hyperparameter tuning. [ABSTRACT FROM AUTHOR]
Copyright of Neural Networks is the property of Pergamon Press - An Imprint of Elsevier Science and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Engineering Source
FullText Text:
  Availability: 0
Header DbId: egs
DbLabel: Engineering Source
An: 174842682
AccessLevel: 6
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Stabilizing machine learning prediction of dynamics: Novel noise-inspired regularization tested with reservoir computing.
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Wikner%2C+Alexander%22">Wikner, Alexander</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> awikner1@umd.edu</i><br /><searchLink fieldCode="AR" term="%22Harvey%2C+Joseph%22">Harvey, Joseph</searchLink><relatesTo>1,2</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Girvan%2C+Michelle%22">Girvan, Michelle</searchLink><relatesTo>1</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Hunt%2C+Brian+R%2E%22">Hunt, Brian R.</searchLink><relatesTo>3</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Pomerance%2C+Andrew%22">Pomerance, Andrew</searchLink><relatesTo>4</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Antonsen%2C+Thomas%22">Antonsen, Thomas</searchLink><relatesTo>1,5</relatesTo> (AUTHOR)<br /><searchLink fieldCode="AR" term="%22Ott%2C+Edward%22">Ott, Edward</searchLink><relatesTo>1,5</relatesTo> (AUTHOR)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="JN" term="%22Neural+Networks%22">Neural Networks</searchLink>. Feb2024, Vol. 170, p94-110. 17p.
– Name: Subject
  Label: Subjects
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Machine+learning%22">Machine learning</searchLink><br /><searchLink fieldCode="DE" term="%22Recurrent+neural+networks%22">Recurrent neural networks</searchLink><br /><searchLink fieldCode="DE" term="%22Biologically+inspired+computing%22">Biologically inspired computing</searchLink><br /><searchLink fieldCode="DE" term="%22Long-Term+Evolution+%28Telecommunications%29%22">Long-Term Evolution (Telecommunications)</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Recent work has shown that machine learning (ML) models can skillfully forecast the dynamics of unknown chaotic systems. Short-term predictions of the state evolution and long-term predictions of the statistical patterns of the dynamics ("climate") can be produced by employing a feedback loop, whereby the model is trained to predict forward only one time step, then the model output is used as input for multiple time steps. In the absence of mitigating techniques, however, this feedback can result in artificially rapid error growth ("instability"). One established mitigating technique is to add noise to the ML model training input. Based on this technique, we formulate a new penalty term in the loss function for ML models with memory of past inputs that deterministically approximates the effect of many small, independent noise realizations added to the model input during training. We refer to this penalty and the resulting regularization as Linearized Multi-Noise Training (LMNT). We systematically examine the effect of LMNT, input noise, and other established regularization techniques in a case study using reservoir computing, a machine learning method using recurrent neural networks, to predict the spatiotemporal chaotic Kuramoto–Sivashinsky equation. We find that reservoir computers trained with noise or with LMNT produce climate predictions that appear to be indefinitely stable and have a climate very similar to the true system, while the short-term forecasts are substantially more accurate than those trained with other regularization techniques. Finally, we show the deterministic aspect of our LMNT regularization facilitates fast reservoir computer regularization hyperparameter tuning. [ABSTRACT FROM AUTHOR]
– Name: AbstractSuppliedCopyright
  Label:
  Group: Ab
  Data: <i>Copyright of Neural Networks is the property of Pergamon Press - An Imprint of Elsevier Science and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.)
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=egs&AN=174842682
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1016/j.neunet.2023.10.054
    Languages:
      – Code: eng
        Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 17
        StartPage: 94
    Subjects:
      – SubjectFull: Machine learning
        Type: general
      – SubjectFull: Recurrent neural networks
        Type: general
      – SubjectFull: Biologically inspired computing
        Type: general
      – SubjectFull: Long-Term Evolution (Telecommunications)
        Type: general
    Titles:
      – TitleFull: Stabilizing machine learning prediction of dynamics: Novel noise-inspired regularization tested with reservoir computing.
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Wikner, Alexander
      – PersonEntity:
          Name:
            NameFull: Harvey, Joseph
      – PersonEntity:
          Name:
            NameFull: Girvan, Michelle
      – PersonEntity:
          Name:
            NameFull: Hunt, Brian R.
      – PersonEntity:
          Name:
            NameFull: Pomerance, Andrew
      – PersonEntity:
          Name:
            NameFull: Antonsen, Thomas
      – PersonEntity:
          Name:
            NameFull: Ott, Edward
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 02
              Text: Feb2024
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 08936080
          Numbering:
            – Type: volume
              Value: 170
          Titles:
            – TitleFull: Neural Networks
              Type: main
ResultId 1