Neural Network Analysis of Psychological Data: A Step-by-Step Guide
Saved in:
| Title: | Neural Network Analysis of Psychological Data: A Step-by-Step Guide |
|---|---|
| Language: | English |
| Authors: | Lingbo Tong, Zhiyong Zhang |
| Source: | Grantee Submission. 2026. |
| Peer Reviewed: | Y |
| Page Count: | 23 |
| Publication Date: | 2026 |
| Sponsoring Agency: | Institute of Education Sciences (ED) |
| Contract Number: | R305D210023 |
| Document Type: | Reports - Evaluative |
| Descriptors: | Artificial Intelligence, Data, Psychological Studies, Network Analysis, Data Analysis, Models, Statistical Inference, Regression (Statistics) |
| DOI: | 10.1080/00273171.2025.2587379 |
| Abstract: | Artificial neural networks (ANN) have attracted increasing attention in the field of psychology. With the availability of software programs, the wide application of ANN becomes possible. However, without a firm understanding of the basics of the ANN, issues can easily arise. This article presents a step-by-step guide for implementing a feed-forward neural network (FNN) on a psychological data set to illustrate the critical steps in building, estimating, and interpreting a neural network model. We start with a concrete example of a basic 3-layer FNN, illustrating the core concepts, the matrix representation, and the whole optimization process. By adjusting parameters and changing the model structure, we examine their effects on model performance. Then, we introduce accessible methods for interpreting model results and making inferences. Through the guide, we hope to help researchers avoid common problems in applying neural network models and machine learning methods in general. [This is the online first version of an article published in "Multivariate Behavioral Research." This work was also supported by the Lucy Family Institute for Data and Society and Notre Dame Global.] |
| Abstractor: | As Provided |
| IES Funded: | Yes |
| Entry Date: | 2026 |
| Accession Number: | ED679891 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwFVgkjWQxHiRY0-igN6HMlRAAAA4jCB3wYJKoZIhvcNAQcGoIHRMIHOAgEAMIHIBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDJ_Trgl4aUIdZPPhSwIBEICBmtzqvihdO3VhV6uxSyi7MGOmovRZC3v-unrolNJ7p6_-AZQ9Ix4ctDSrrxxtagnKI_hOtwi95NrO3AMxvCiPnkbJBKRLI5q3-sWXluUwOdGXbIVFxHhmbVDd0KgW2RTw0Z5z5zDHILkmtUNFH0_UnOsS4HkkTRAzRQB9r-HXABgk5-6Qka18EikZk9kPqLQUE4XOB2L2zqNtWMA= Text: Availability: 1 Value: <anid>AN0193318148;3zw01mar.26;2026Apr30.06:29;v2.2.500</anid> <title id="AN0193318148-1">Neural Network Analysis of Psychological Data: A Step-by-Step Guide </title> <p>Artificial neural networks (ANN) have attracted increasing attention in the field of psychology. With the availability of software programs, the wide application of ANN becomes possible. However, without a firm understanding of the basics of the ANN, issues can easily arise. This article presents a step-by-step guide for implementing a feed-forward neural network (FNN) on a psychological data set to illustrate the critical steps in building, estimating, and interpreting a neural network model. We start with a concrete example of a basic 3-layer FNN, illustrating the core concepts, the matrix representation, and the whole optimization process. By adjusting parameters and changing the model structure, we examine their effects on model performance. Then, we introduce accessible methods for interpreting model results and making inferences. Through the guide, we hope to help researchers avoid common problems in applying neural network models and machine learning methods in general.</p> <p>Keywords: Feed-forward neural network (FNN); machine learning; psychological data sets; stochastic gradient descent (SGD); hyperparameter optimization</p> <hd id="AN0193318148-2">Introduction</hd> <p>An <emph>artificial neural network</emph> (ANN), also referred to simply as a <emph>neural network</emph>, is a computational model inspired by the structure and functioning of the human brain (Bishop &amp; Nasrabadi, [<reflink idref="bib10" id="ref1">10</reflink>]). It consists of interconnected units (or "neurons") organized into layers that transform and pass information forward through the network. Compared with the traditional statistical models such as linear regression, neural networks are particularly useful for modeling complex, nonlinear relationships in data.</p> <p>Among the simplest forms of neural networks is the <emph>feed-forward neural network</emph> (FNN), where information flows in one direction, from the input layer through one or more hidden layers to the output layer, without loops or cycles. Even with a single hidden layer, FNNs are capable of approximating a wide range of functions, making them a foundational building block in modern machine learning (Schmidhuber, [<reflink idref="bib73" id="ref2">73</reflink>]).</p> <p>When a neural network contains more than one hidden layer, it is commonly referred to as a <emph>deep learning</emph> (DL) model. The "deep" in deep learning refers to the depth of the architecture (multiple layers of transformations between input and output) that allows the model to learn complex representations of the data (LeCun et al., [<reflink idref="bib42" id="ref3">42</reflink>]). Deep learning builds on the same foundational principles as simpler neural networks, but extends them in depth and capacity, making them powerful in solving complex tasks in data-rich environments.</p> <p>Deep learning has become an indispensable tool across various research disciplines (LeCun et al., [<reflink idref="bib42" id="ref4">42</reflink>]). Their ability to learn from and make predictions or decisions based on data has been widely acknowledged and harnessed in multiple fields, ranging from downstream areas such as natural language processing and computer vision (Goldberg, [<reflink idref="bib26" id="ref5">26</reflink>]; Khan et al., [<reflink idref="bib40" id="ref6">40</reflink>]), to cross-cutting disciplines, for example, computational neuroscience and bioinformatics (Min et al., [<reflink idref="bib52" id="ref7">52</reflink>]; Richards et al., [<reflink idref="bib66" id="ref8">66</reflink>]). In psychological research, deep learning has been applied to assisting psychiatrists and psychologists in various tasks such as mental disorder diagnosis (Iyortsuun et al., [<reflink idref="bib35" id="ref9">35</reflink>]; Su et al., [<reflink idref="bib75" id="ref10">75</reflink>]), suicide risk detection (Malhotra &amp; Jindal, [<reflink idref="bib48" id="ref11">48</reflink>]; Tadesse et al., [<reflink idref="bib78" id="ref12">78</reflink>]), and personality assessment (Abdurahman et al., [<reflink idref="bib2" id="ref13">2</reflink>]).</p> <p>The exponential growth of the internet has facilitated the collection of massive data sets, combined with the advent of computing resources, granting scientists opportunities to create and distribute large-scale deep learning models (Abadi et al., [<reflink idref="bib1" id="ref14">1</reflink>]; Kaddour et al., [<reflink idref="bib38" id="ref15">38</reflink>]). However, the current research landscape seems oriented toward these large-scale models and extensive data sets, leaving a gap in exploring the efficacy of portable models in smaller, lab-collected data sets, standard in social sciences. This issue is particularly pronounced within the field of psychology, where researchers often struggle to adopt neural-network-based methods in their own research due to lack access to large data sets, computational resources, or training in machine learning methods (Yarkoni &amp; Westfall, [<reflink idref="bib84" id="ref16">84</reflink>]).</p> <p>While there are primers designed to introduce psychologists to deep learning (Urban and Gates ([<reflink idref="bib79" id="ref17">79</reflink>]; Orrù et al. [<reflink idref="bib56" id="ref18">56</reflink>]), many of these resources assume familiarity with machine learning concepts or focus on high-level overviews rather than step-by-step model building and training. The gap becomes particularly evident when considering applied researchers with a background in regression modeling but limited experience with neural network implementations. Therefore, a tutorial featuring clear and concrete examples focusing on these approaches will be beneficial, as it could assist psychologists in comprehending the fundamental mechanism of neural network models and their usage.</p> <p>Addressing this knowledge gap, the present article offers a step by step tutorial for implementing neural networks in psychological research, through the analysis of the data from the Advanced Cognitive Training for Independent and Vital Elderly (ACTIVE) study (Jobe et al., [<reflink idref="bib37" id="ref19">37</reflink>]). Particularly, we show how to apply a fundamental 3-layer FNN to the data set and explain the neuron update processes involved. Subsequently, we discuss model optimization, examining how various parameter adjustments and model structures can influence the network's performance. Additionally, we explore methods for interpreting model results and making inferences. Finally, we conclude with an overview on the development of deep learning models in recent years, offering a primer for psychologists aiming to incorporate neural networks into their research. The tutorial should provide researchers with a clear understanding on how the methods work and can help researchers in conditions ranging from building their own models to applying existing techniques such as ChatGPT.</p> <hd id="AN0193318148-3">Data set</hd> <p>This article's data are drawn from the ACTIVE data set (Jobe et al., [<reflink idref="bib37" id="ref20">37</reflink>]). Originating from a large-scale study conducted from 1999 to 2001 in the United States, the ACTIVE data set involved six field sites and aimed to evaluate the impact of three cognitive interventions (focused on memory, executive reasoning, and processing speed) on daily cognitive activities in older adults. The study included 2,832 participants, with assessments conducted at baseline, post-training, and annually thereafter. A subset of 11 variables from the ACTIVE study is presented in Table 1, which will be used in our illustration. After excluding records with empty or invalid values, the data set comprises 1,573 participants.[<reflink idref="bib1" id="ref21">1</reflink>]</p> <p>Table 1. Variable names and descriptions in the ACTIVE dataset.</p> <p> <ephtml> &lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;Variable&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Meaning&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Occasion&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody valign="top"&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;age&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Participant age&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Baseline&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;edu&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Years of education&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Baseline&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;sex&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;1: male; 2: Female&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Baseline&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;booster&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Whether received booster training&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Baseline&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;reason&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Reasoning ability&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Baseline&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;ufov&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Useful field of view variable&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Baseline&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;mmse&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Mini-mental state examination total score&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Baseline&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;hvltt&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Hopkins Verbal Learning Test total score at time 1&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Baseline&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;hvltt2&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Hopkins Verbal Learning Test total score at time 2&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Post-training&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;hvltt3&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Hopkins Verbal Learning Test total score at time 3&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;One year later&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;hvltt4&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Hopkins Verbal Learning Test total score at time 4&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Two years later&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>In the examples throughout this article, we use hvltt4 as the model outcome. Our primary research question is how accurately this outcome can be predicted using the remaining variables in the data set. We use this predictive task to demonstrate neural network training techniques and hyperparameter tuning strategies.</p> <hd id="AN0193318148-4">Feed-Forward neural network</hd> <p>A feed-forward neural network (FNN), also known as a multilayer perceptron (MLP), represents a basic building block for more complex neural network architectures. In an FNN, information flows in one direction, from the input layer through one or more hidden layers to the output layer, without any feedback loops (Sazli, [<reflink idref="bib72" id="ref22">72</reflink>]). Each layer consists of interconnected computational units (often called "neurons" or "nodes") that receive inputs, apply transformations using learned parameters (weights and biases), and pass outputs forward to subsequent layers. In this paper, we use the term "neurons" to refer specifically to these computational units or nodes, rather than to individual parameters.</p> <p>The simplest FNN contains three layers: an input layer, a hidden layer, and an output layer. Figure 1 illustrates such structure for a single observation <emph>i</emph>. In practice, we typically feed multiple observations simultaneously into the FNN <emph>via</emph> matrix multiplication.</p> <p>Graph: Figure 1. Illustration of a three-layer FNN for a single observation i. Since the task involves predicting a continuous outcome, the output layer uses an identity activation function (not explicitly shown here).</p> <p>Mathematically, the model can be defined as</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;munder&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;munder accentunder="true"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;&amp;#981;&lt;/mi&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn mathvariant="bold"&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#9183;&lt;/mo&gt;&lt;/munder&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext&gt;Hidden layer output&lt;/mtext&gt;&lt;/mrow&gt;&lt;/munder&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn mathvariant="bold"&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib1" id="ref23">1</reflink>)</p> <p>Here,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is the input data matrix,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> denotes the weight matrix connecting the input layer and the hidden layer, and</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is the corresponding row vector of bias parameters for the hidden layer.</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;&amp;#981;&lt;/mi&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> represents an activation function. Similarly,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> denotes the weight matrix connecting the hidden layer to the output layer, and</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is the associated row vector of biases for the output layer.</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo mathvariant="bold"&gt;&amp;#8257;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> indicates the predicted outcomes of the model in the output layer. The actual outcomes are represented by</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> observed in the data. We now explain each component in subsequent subsections, using the ACTIVE data example. Suppose in the example, we want to predict the verbal test score at time 4 (hvltt4) using age, years of education, and gender.</p> <hd id="AN0193318148-5">Input layer</hd> <p>The input layer of the FNN is represented as an</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> matrix, with <emph>n</emph> denoting the number of participants and <emph>p</emph> signifying the number of input variables or predictors. As an example, we consider three predictors (</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;3&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> ): <emph>age</emph>, <emph>years of education</emph>, and <emph>sex</emph>. For illustration, we let</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;5&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> here. The resulting raw input data matrix, denoted as</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;raw&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> appears as follows with the collected data in the ACTIVE study:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;raw&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;11&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;12&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;13&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;21&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;22&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;23&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;31&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;32&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;33&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;41&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;42&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;43&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;51&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;52&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;53&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;65&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;12&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;70&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;13&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;83&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;12&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;72&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;16&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;66&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;13&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib2" id="ref24">2</reflink>)</p> <p>Here,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> indicates the value for the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;th&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> participant of the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;j&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;th&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> predictor. For instance, the first row in the input matrix represents a record from a 65-year-old female participant with 12 years of education. The data in the input layer are known and observed.</p> <p>A common preprocessing step is to normalize the input features by subtracting the mean and dividing by the standard deviation, that is, standardizing the data to have zero mean and unit variance (LeCun et al., [<reflink idref="bib44" id="ref25">44</reflink>]). This technique ensures that all input features are on a similar scale, preventing any single feature from dominating the learning process due to its larger variance (Zheng &amp; Casari, [<reflink idref="bib87" id="ref26">87</reflink>]). Furthermore, it helps gradient-based optimization algorithms converge faster and more effectively during training (Goodfellow et al., [<reflink idref="bib27" id="ref27">27</reflink>]). After normalization, the standardized input data matrix, denoted as</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> becomes:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.4476&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.6565&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.5421&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.548&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.2731&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.8447&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.7909&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.6565&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.5421&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.1882&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.8772&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.8447&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.2677&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.2731&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.5421&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib3" id="ref28">3</reflink>)</p> <hd id="AN0193318148-6">Hidden layer</hd> <p>Hidden layers in FNNs can be conceptually compared to latent variables in structural equation modeling. While latent variables represent the theoretical constructs specified by the researcher, hidden layers learn the data-driven representations optimized for predictive accuracy.</p> <p>A hidden layer in an FNN consists of a set of neurons between the input layer and the output layer. Each neuron in a hidden layer transforms the values from the previous layer with a weighted linear summation followed by a linear or nonlinear activation function.</p> <p>The layer transformation takes the form</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi&gt;&amp;#981;&lt;/mi&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn mathvariant="bold"&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> where</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is an</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> matrix representing the input data from the previous layer,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is a</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;p&lt;/mi&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;h&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> weight matrix denoting the linear transformation applied to the input data, and</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is defined as a</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;h&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> row vector. Here,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn mathvariant="bold"&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is an</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> column vector of ones, and the multiplication</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn mathvariant="bold"&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> ensures the bias vector is correctly added (or <emph>broadcasted</emph>) to each row of the matrix product</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> The activation function</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;&amp;#981;&lt;/mi&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is then applied element-wise, yielding the transformed hidden layer output matrix</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> which has dimensions</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;h&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> </p> <p>For instance, if</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;h&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;4&lt;/mn&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> that is, the hidden layer comprises 4 neurons, the weights and biases would be structured as follows:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;11&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;12&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;13&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;14&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;21&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;22&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;23&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;24&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;31&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;32&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;33&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;34&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mtext /&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mn&gt;3&lt;/mn&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mn&gt;4&lt;/mn&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib4" id="ref29">4</reflink>)</p> <p>where</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> denotes the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;th&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> weight value for the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;j&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;th&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> neuron of the hidden layer, and</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;j&lt;/mi&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> denotes the bias associated with the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;j&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;th&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> neuron of that layer. During the linear summation, the input data matrix <bold>X</bold> is multiplied by the weight matrix</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> yielding a matrix of dimensions</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;h&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> Subsequently, the bias vector</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is added to each row of the product matrix, a process called <emph>broadcasting</emph>.</p> <p>The linear summation is typically followed by a nonlinear activation function, although a linear function can also be used, enabling the network to capture complex patterns and relationships within the data. The activation function, denoted as</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;&amp;#981;&lt;/mi&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> will be applied to each element in the output matrix of the linear transformation.</p> <p>Different nonlinear functions could be used as activation functions in FNN. One widely applied function is the Sigmoid function, characterized by its S-shaped curve, transforming input values into a value between 0 and 1. The function is represented as:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtext&gt;Sigmoid&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;e&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib5" id="ref30">5</reflink>)</p> <p>Another popular activation function is the Rectified Linear Unit (ReLU) function, a piecewise linear function that outputs the input value if it's positive and zero if it's negative. Formally,</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtext&gt;ReLU&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mtext mathvariant="normal"&gt;max&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib6" id="ref31">6</reflink>)</p> <p>ReLU has become a favored choice in FNNs due to its computational efficiency and ability to mitigate the vanishing gradient problem during training (Agarap, [<reflink idref="bib3" id="ref32">3</reflink>]; Yu &amp; Zhu, [<reflink idref="bib85" id="ref33">85</reflink>]). In this paper, we will consistently utilize ReLU in all our experiments. It is worth noting, however, that the Sigmoid function is commonly preferred in binary classification tasks, where its output range of 0 to 1 naturally represents a probability. For readers seeking a more in-depth comparison of activation functions and their roles in deep learning, we recommend Urban and Gates ([<reflink idref="bib79" id="ref34">79</reflink>]).</p> <p>Following the activation function, we obtain the output matrix <bold>H</bold>, maintaining dimensions</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;h&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> representing the values post-transformation of the input data.</p> <hd id="AN0193318148-7">Output layer</hd> <p>The output layer transforms the values derived from the last hidden layer, yielding the final output. The choice of activation function for the output layer depends on the nature of the prediction task. For problems where the target variable is continuous, it is common not to have an activation function, or to use an identity activation function, in the output layer. Therefore, this transformation is mathematically represented as</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn mathvariant="bold"&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> Here, the weight matrix</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> and the bias vector</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> have dimensions of</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;h&lt;/mi&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;q&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> and</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;q&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> respectively, with <emph>q</emph> denoting the number of output variables. The number of output variables depends on the structure of observed data and research questions, that is, how many outcome variables to predict. In the context of this example, with the single variable <emph>hvltt4</emph> specified as the output,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;q&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> determining the weights and biases as follows:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;11&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;21&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;31&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;w&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;41&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mtext /&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib7" id="ref35">7</reflink>)</p> <p>Consequently, the final output matrix</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> possesses the dimensions of</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi&gt;q&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> Each element</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> within</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> signifies the predicted outcome for the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;th&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> participant concerning the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;j&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;th&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> dependent variable. In instances where</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;q&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> the final output would be:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;11&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;21&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;31&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;41&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;51&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib8" id="ref36">8</reflink>)</p> <p>Considering all the weight matrices and bias vectors combined, the total number of parameters to estimate in a three-layer FNN is given by the formula</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mo&gt;#&lt;/mo&gt;&lt;mtext&gt;params=&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mtext&gt;p + 1&lt;/mtext&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mtext&gt;h+&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mtext&gt;h + 1&lt;/mtext&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mi mathvariant="normal"&gt;q&lt;/mi&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib9" id="ref37">9</reflink>)</p> <p>For our specific case, this calculation becomes</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;3&lt;/mn&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mn&gt;4&lt;/mn&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;4&lt;/mn&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;21&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> parameters. The following section explains how the model iteratively updates its parameters to estimate their optimal values.</p> <hd id="AN0193318148-8">Model training and evaluation</hd> <p>In traditional regression models, estimating model parameters usually means finding regression coefficients that minimize the sum of squared errors. In machine learning, this process is referred to as <emph>model training</emph>, which is conceptually analogous to parameter estimation. During training, the model learns from a data set by iteratively adjusting its parameters to minimize prediction errors. <emph>Model evaluation</emph>, in turn, assesses how well a model's predictions align with the actual outcomes.</p> <hd id="AN0193318148-9">Data split</hd> <p>Before beginning parameter estimation (training), we set aside a portion of the data set for evaluation purposes. This approach differs somewhat from traditional multivariate modeling techniques (e.g., structural equation modeling, SEM), where researchers typically utilize the entire data set to estimate parameters and then evaluate model fit using fit indices such as BIC, RMSEA, or CFI. In contrast, machine learning emphasizes the model's predictive performance on unseen data. Specifically, the primary goal of machine learning modeling is generalizability: the ability of a model to accurately predict outcomes on new data beyond the sample used for parameter estimation. Using separate subsets of the data to train, validate, and test the model ensures that evaluation metrics truly reflect predictive accuracy rather than merely describing model fit to the training sample.</p> <p>A common practice in machine learning is to divide the original data set into different subsets for training, validation (optional), and testing. The training set is used for the learning process. The validation set helps to fine-tune the hyperparameters, determine the appropriate time to stop the training process, and facilitate the model selection. The test set is used for final evaluation.</p> <p>In this tutorial, we employ a two-tiered splitting strategy: Initially, we randomly selected 15% of the entire data set as the test set. Subsequently, from the remaining data, we randomly take 15% as the validation set. This partitioning results in training, validation, and test sets comprising approximately 72.25%, 12.25%, and 15% of the original data set, respectively.</p> <hd id="AN0193318148-10">Training</hd> <p>Training an FNN is the process of finding the weights and biases in a neural network. It involves several key steps, including parameter initialization, forward propagation, calculation of the loss function, and backpropagation.</p> <hd id="AN0193318148-11">Parameter initialization</hd> <p>The process begins with <emph>initialization</emph>, where the network's parameters (weights and biases) are assigned selected, often random, initial values. For example, for each linear layer that receives <emph>a</emph> inputs from the previous layer, its weights and biases can be initialized from a uniform distribution</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;U&lt;/mi&gt;&lt;mo stretchy="true"&gt;(&lt;/mo&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;msqrt&gt;&lt;mi&gt;k&lt;/mi&gt;&lt;/msqrt&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;msqrt&gt;&lt;mi&gt;k&lt;/mi&gt;&lt;/msqrt&gt;&lt;mo stretchy="true"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> where</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;k&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mi&gt;a&lt;/mi&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> through random number generation. Note that <emph>a</emph> refers to the number of inputs to a given layer, which may differ from <emph>p</emph>, the number of input features initially provided by the researcher at the input layer.</p> <p>In the context of our example, the initialized parameter matrices appear as follows:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0387&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.9131&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0134&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.4025&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.8408&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.7913&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0134&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0391&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.3027&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.7905&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.7644&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.1751&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mtext /&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.4067&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.5735&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.3693&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.0427&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib10" id="ref38">10</reflink>)</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.0246&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.7404&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.5110&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.0512&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mtext /&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.0290&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib11" id="ref39">11</reflink>)</p> <hd id="AN0193318148-12">Forward propagation</hd> <p>During <emph>forward propagation</emph>, input data traverse the network, producing predictions. First, the input data (the example data in Equation (<reflink idref="bib2" id="ref40">2</reflink>)) goes through the linear summation in the hidden layer, which leads to:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn mathvariant="bold"&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.0667&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;2.8433&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0169&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.6092&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0567&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.1683&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.7904&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.1558&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.1921&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.1138&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0603&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.6943&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.8966&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.4071&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.7702&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.2557&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.7513&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;2.3756&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0244&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.5518&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib12" id="ref41">12</reflink>)</p> <p>Then, the activation function is applied, here a ReLU, to get:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mtext&gt;ReLU&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.0667&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;2.8433&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0169&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.6092&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0567&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.1921&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0603&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.7513&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;2.3756&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0244&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.5518&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib13" id="ref42">13</reflink>)</p> <p>And finally, the output layer can be obtained as:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn mathvariant="bold"&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;3.2497&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.0871&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.2196&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.0290&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;2.5735&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib14" id="ref43">14</reflink>)</p> <hd id="AN0193318148-13">Calculating the loss function</hd> <p>Now, we have finished the first round of forward propagation, resulting in a set of predictions</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> The difference between these predictions and actual values is quantified by a <emph>loss function</emph>, reflecting how well the network is performing at the current stage in the training process. One of the most popular loss functions for continuous data is the mean squared error (MSE; Gareth et al., [<reflink idref="bib24" id="ref44">24</reflink>]), which can be defined as:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtext&gt;MSE&lt;/mtext&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;nq&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;munderover&gt;&lt;mo&gt;&amp;#8721;&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;/munderover&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;munderover&gt;&lt;mo&gt;&amp;#8721;&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;j&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mi&gt;q&lt;/mi&gt;&lt;/munderover&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib15" id="ref45">15</reflink>)</p> <p>where <emph>n</emph> is the sample size,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is the observed value of the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;j&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;th&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> dependent variable for the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;th&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> participant, and</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is the corresponding predicted value generated from the neural network.</p> <p>Given the observed values of the outcome variable</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;34&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;27&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;27&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;26&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;27&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib16" id="ref46">16</reflink>)</p> <p>and the prediction in the output layer in Equation (<reflink idref="bib14" id="ref47">14</reflink>), the MSE is then calculated as</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtext&gt;MSE&lt;/mtext&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;5&lt;/mn&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;munderover&gt;&lt;mo&gt;&amp;#8721;&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mn&gt;5&lt;/mn&gt;&lt;/munderover&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;178.79&lt;/mn&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib17" id="ref48">17</reflink>)</p> <p>which is the average squared difference between the predicted values and the actual values in the data set. Taking the square root of this value, we obtain a Root Mean Square Error (RMSE) of 13.37. This metric quantifies how far the model's predictions are from the observed values on average after the first round of forward propagation.</p> <hd id="AN0193318148-14">Gradient descent and backpropagation</hd> <p>In traditional statistical modeling, such as maximum-likelihood estimation (MLE), parameters are estimated by maximizing the likelihood of the observed data under a specified model. To maximize the likelihood function, iterative algorithms are often used. Similarly, neural networks are typically trained using iterative optimization algorithms that minimize a loss function directly, without relying on an explicit probabilistic model.</p> <p>One of the most widely used optimization techniques in this context is <emph>gradient descent</emph>. Gradient descent is an iterative optimization method that updates model parameters in the direction opposite to the gradient of the loss function. The gradient indicates how the loss changes with respect to each parameter; thus, moving in the negative gradient direction gradually reduces the loss. In its basic form, gradient descent goes over the entire training data set at each iteration, which can be computationally expensive, especially for large data sets.</p> <p>Mathematically, the update rule for gradient descent is given by:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi mathvariant="bold-italic"&gt;&amp;#952;&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi mathvariant="bold-italic"&gt;&amp;#952;&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi&gt;&amp;#951;&lt;/mi&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi mathvariant="bold-italic"&gt;&amp;#952;&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib18" id="ref49">18</reflink>)</p> <p>where</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold-italic"&gt;&amp;#952;&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> represents a vector of all the model's parameters, <emph>t</emph> is the iteration index,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;&amp;#951;&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is the learning rate—a positive scalar determining the step size in the direction opposite to the gradient, and</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi mathvariant="bold-italic"&gt;&amp;#952;&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is the gradient of the model loss <emph>l</emph> with respect to</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold-italic"&gt;&amp;#952;&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> </p> <p>In practice, we typically use a stochastic variant of this algorithm, known as <emph>Stochastic Gradient Descent (SGD)</emph>. Instead of going over the entire data set at each iteration, SGD takes a small, randomly selected batch of training examples. This approach is computationally more efficient and introduces helpful noise into the optimization process.</p> <p>To compute these gradients efficiently in a neural network, we use the <emph>backpropagation</emph> algorithm. Backpropagation is a recursive method for efficiently computing the gradient of the loss function with respect to each parameter in the network. It works by propagating the error signal from the output layer backward through the network, layer by layer, computing the contribution of each parameter to the total error.</p> <p>In short, gradient descent defines the optimization strategy, SGD offers a practical and scalable implementation of that strategy, and backpropagation provides an efficient way to compute the required gradients.</p> <p>We now illustrate how to derive these gradients step by step using our example data set, starting with the mean squared error (MSE) loss function:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mtext&gt;MSE&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8727;&lt;/mo&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;munderover&gt;&lt;mo&gt;&amp;#8721;&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;i&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;/munderover&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;munderover&gt;&lt;mo&gt;&amp;#8721;&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;j&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mi&gt;q&lt;/mi&gt;&lt;/munderover&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib19" id="ref50">19</reflink>)</p> <p>Here, <emph>l</emph> is a scalar denoting the average squared loss. Note that we have modified the standard MSE loss to MSE* for computational ease during differentiation. Firstly, we omit the constant coefficient</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;nq&lt;/mtext&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> since both <emph>n</emph> (the number of samples) and <emph>q</emph> (the number of output variables per sample) are constants and do not affect the optimization process. Secondly, we introduce a factor of</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;/&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> This adjustment is made because the squared term in MSE leads to a factor of 2 when we take its derivative. By incorporating the</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;/&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> factor, the 2 will be canceled out when differentiating, simplifying the equations.</p> <p>The partial derivative of the loss function with respect to each predicted value</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;munderover&gt;&lt;mo&gt;&amp;#8721;&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;k&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;/munderover&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;munderover&gt;&lt;mo&gt;&amp;#8721;&lt;/mo&gt;&lt;mrow&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mi&gt;q&lt;/mi&gt;&lt;/munderover&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;kl&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;kl&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mrow&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mtext mathvariant="italic"&gt;ij&lt;/mtext&gt;&lt;/mrow&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib20" id="ref51">20</reflink>)</p> <p>In matrix form, this gradient is a matrix of the same dimensions as</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo mathvariant="bold"&gt;&amp;#8257;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> where each element is the derivative with respect to the corresponding element in</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo mathvariant="bold"&gt;&amp;#8257;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> Thus, it can also be denoted as:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;37.2497&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;27.0871&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;28.2196&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;26.0290&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;29.5735&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib21" id="ref52">21</reflink>)</p> <p>The subsequent steps of computing the gradients are guided by the chain rule of derivatives, formally,</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;z&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;z&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;y&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;x&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> To illustrate clearly how the chain rule operates in matrix form, we first consider a general case explicitly. Suppose we have a scalar function <emph>g</emph> that depends on the matrix</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;U&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> where the matrix</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;U&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> itself is defined as a linear transformation of another matrix</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;V&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> Specifically, let</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;U&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mtext mathvariant="bold"&gt;VW&lt;/mtext&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;C&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> with</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;C&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> representing a bias matrix. Under these conditions, the chain rule in matrix calculus notation is expressed as follows:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;g&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi&gt;f&lt;/mi&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;U&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;mo /&gt;&lt;mi mathvariant="bold"&gt;U&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mtext mathvariant="bold"&gt;VW&lt;/mtext&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;C&lt;/mi&gt;&lt;mi /&gt;&lt;mo&gt;&amp;#8658;&lt;/mo&gt;&lt;mi /&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;g&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;V&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;g&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;U&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;mo /&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;g&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;V&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;g&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;U&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib22" id="ref53">22</reflink>)</p> <p>This formulation becomes especially handy when deriving gradients in deep learning since our loss function typically outputs a scalar value, while the model parameters are in matrix or vector form. For instance, since</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> the gradient of the loss <emph>l</emph> with respect to the weight matrix</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;97.13&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;176.1658&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;3.0513&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;39.0105&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib23" id="ref54">23</reflink>)</p> <p>Likewise, the gradient of the loss <emph>l</emph> with respect to the hidden layer output</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;38.1661&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;27.5797&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;19.0346&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.9072&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;27.7534&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;20.0553&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;13.8415&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.3869&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;28.9138&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;20.8938&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;14.4202&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.4448&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;26.6693&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;19.2719&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;13.3008&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.3327&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;30.301&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;21.8962&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;15.112&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.5142&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib24" id="ref55">24</reflink>)</p> <p>The gradient of <emph>l</emph> with respect to</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> can be derived in the same way. Since</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is actually</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn mathvariant="bold"&gt;1&lt;/mn&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> with 1 denoting a column vector of ones in strict matrix algebra notations, the gradient concerning</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> is given by:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mo&gt;&amp;#8942;&lt;/mo&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mo&gt;&amp;#8942;&lt;/mo&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;148.1589&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib25" id="ref56">25</reflink>)</p> <p>Next, we come to the output of the first linear transformation</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;/mrow&gt;&lt;mtext&gt;.&lt;/mtext&gt;&lt;/math&gt; </ephtml> Given</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mtext&gt;ReLU&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> and the derivative of</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;u&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mtext&gt;ReLU&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;mtext&gt;:&lt;/mtext&gt;&lt;/math&gt; </ephtml> </p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mi&gt;&amp;#948;&lt;/mi&gt;&lt;mi&gt;u&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;&amp;#948;&lt;/mi&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mtext&gt;sign&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mtext mathvariant="normal"&gt;max&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;{&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;&amp;#62;&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;&amp;#8804;&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib26" id="ref57">26</reflink>)</p> <p>we can apply the chain rule to get:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;H&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mtext&gt;sign&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mtext mathvariant="normal"&gt;max&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib27" id="ref58">27</reflink>)</p> <p>Then, by</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;+&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> we get</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;X&lt;/mi&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mtext&gt;sign&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mtext mathvariant="normal"&gt;max&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib28" id="ref59">28</reflink>)</p> <p>and</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mo&gt;&amp;#8942;&lt;/mo&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mo&gt;&amp;#8942;&lt;/mo&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mi&gt;n&lt;/mi&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mover accent="true"&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;&amp;#770;&lt;/mo&gt;&lt;/mover&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Y&lt;/mi&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;mi&gt;T&lt;/mi&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#183;&lt;/mo&gt;&lt;mtext&gt;sign&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mtext mathvariant="normal"&gt;max&lt;/mtext&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib29" id="ref60">29</reflink>)</p> <p>Plugging in numerical values, we have:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi mathvariant="bold"&gt;Z&lt;/mi&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;38.1661&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;27.5797&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;19.0346&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.9072&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;27.7534&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;28.9138&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;14.4202&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;30.301&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;21.8962&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;15.112&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;1.5142&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib30" id="ref61">30</reflink>)</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;60.3886&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;42.3639&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;30.7332&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;2.6078&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;35.5585&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;24.945&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;18.0965&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;1.5355&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;46.7778&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;32.8156&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;23.8063&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;2.02&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib31" id="ref62">31</reflink>)</p> <p>and</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;148.336&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;104.0608&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;75.4916&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;6.4057&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib32" id="ref63">32</reflink>)</p> <p>Now that we have derived gradients for all model parameters and hidden layer outputs, we can update the model parameters following (<reflink idref="bib18" id="ref64">18</reflink>) to be:</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi&gt;&amp;#951;&lt;/mi&gt;&lt;mo&gt;&amp;#8727;&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib33" id="ref65">33</reflink>)</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi&gt;&amp;#951;&lt;/mi&gt;&lt;mo&gt;&amp;#8727;&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib34" id="ref66">34</reflink>)</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi&gt;&amp;#951;&lt;/mi&gt;&lt;mo&gt;&amp;#8727;&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib35" id="ref67">35</reflink>)</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mi&gt;&amp;#951;&lt;/mi&gt;&lt;mo&gt;&amp;#8727;&lt;/mo&gt;&lt;mrow&gt;&lt;mfrac&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mi&gt;l&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8706;&lt;/mo&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;b&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/mfrac&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib36" id="ref68">36</reflink>)</p> <p>For example, when</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;&amp;#951;&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;0.0001&lt;/mn&gt;&lt;/mrow&gt;&lt;mtext&gt;,&lt;/mtext&gt;&lt;/math&gt; </ephtml> </p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msubsup&gt;&lt;mrow&gt;&lt;mi mathvariant="bold"&gt;W&lt;/mi&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mi&gt;t&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;/mrow&gt;&lt;mrow&gt;&lt;mo stretchy="false"&gt;(&lt;/mo&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo stretchy="false"&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;/msubsup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> becomes</p> <p>Graph</p> <p> <ephtml> &lt;math display="block" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr columnalign="left"&gt;&lt;mtd columnalign="left"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo stretchy="true"&gt;[&lt;/mo&gt;&lt;mrow&gt;&lt;mtable&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0449&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.9086&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0103&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.4022&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.8372&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mo&gt;&amp;#8722;&lt;/mo&gt;&lt;mn&gt;0.7887&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0116&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.0393&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;mtr&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.3075&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.7939&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.762&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;mtd&gt;&lt;mrow&gt;&lt;mn&gt;0.1753&lt;/mn&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;mo stretchy="true"&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;.&lt;/mo&gt;&lt;/mrow&gt;&lt;/mtd&gt;&lt;/mtr&gt;&lt;/mtable&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> (<reflink idref="bib37" id="ref69">37</reflink>)</p> <p>This concludes the first round of forward and backward propagation. The process can be repeated. Concrete values of gradients and updated parameters during the first three rounds and the associated Python code for reproducing these values can be found in the code repository.</p> <hd id="AN0193318148-15">Batches, epochs, and early stopping</hd> <p>The five samples or participants used in the example are called one <emph>batch</emph> of data, many times also referred to as <emph>minibatch</emph> (Masters &amp; Luschi, [<reflink idref="bib50" id="ref70">50</reflink>]). In neural network training, a minibatch is a subset of the training data set used for a single step of gradient calculation and weight updating. Compared with computing gradients over the entire data set, SGD with minibatches is more computational efficient, especially when working with large data sets, as it reduces the amount of data that must be processed at each step. It also helps stabilize and accelerate learning by introducing small noise into the gradient estimates. Because each minibatch is a different random sample, the gradient it produces is just a quick but imperfect guess of the true gradient across all data. This variation between steps creates a "noisy" path toward the optimal solution, which helps the model escape local minima and explore the error surface more effectively. One helpful analogy is to imagine a ball rolling down a bumpy landscape; the random jolts can provide the necessary kick to escape a small divot and continue toward a deeper valley.</p> <p>The number of training samples in one batch is called <emph>batch size</emph>. The neural network updates its parameters with each batch processed, and thus, the batch size can influence both the duration of training and the performance of the model.</p> <p>If the total number of training samples is not evenly divisible by the batch size, the final batch of each epoch will contain fewer samples than the specified batch size. This smaller batch is still used to compute gradients and update the model. Most deep learning frameworks handle this automatically. Alternative strategies include padding (i.e., adding synthetic or duplicated samples to the final batch to match the expected batch size) or discarding the last few samples, but using the remainder batch is the standard practice. For more comprehensive discussion on batch and SGD algorithm, see Goodfellow et al. ([<reflink idref="bib27" id="ref71">27</reflink>], Chapters 8.1–8.3).</p> <p>Once the network has processed all the data in the training data set, batch by batch, this completes one training iteration, typically called an <emph>epoch</emph>. In the traning stage, the model repeatedly cycles through the full data set across multiple epochs. Reusing the data across epochs allows the model to gradually refine its parameter estimates through many small adjustments, rather than attempting to learn everything in a single pass. Throughout these epochs, the loss should ideally decrease, signifying that the network is learning the data. Training usually continues until the loss reaches an acceptable level or stops decreasing significantly, indicating that the model has converged.</p> <p>Both training and validation sets are involved in the training process. Specifically, after each training epoch, we test the model on the validation set without updating the parameters. This approach gives us insight into how well the model is learning over time. For instance, we can terminate the training if the model's loss on the validation set stops decreasing (or decreases by less than a tiny amount) over a number of consecutive epochs such as 20. This technique, known as <emph>early stopping</emph>, prevents overfitting by stopping the training before the total number of epochs reaches a pre-defined maximum, which is typically set to a large value. Figure 2 illustrates how the FNN's training and validation loss changed while being trained with a learning rate of 0.0001 in SGD and a batch size of 64. The training loss curve appears smooth because we performed gradient descent on the training set that has more participants, while the validation loss curve has more fluctuations. The curve flattened out around the 80th training epoch and was terminated around the 100th epoch, where early stopping was triggered.</p> <p>PHOTO (COLOR): Figure 2. Change of MSE loss with the number of epochs during the training stage.</p> <hd id="AN0193318148-16">Evaluation</hd> <p>After training the FNN, our next step is to evaluate its performance on the test set. Depending on the task and focus, different metrics can be used to evaluate the performance. In this paper, we continue to use MSE to assess the performance of model predictions. Other popular metrics include Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;msup&gt;&lt;mrow&gt;&lt;mi&gt;R&lt;/mi&gt;&lt;/mrow&gt;&lt;mn&gt;2&lt;/mn&gt;&lt;/msup&gt;&lt;/mrow&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> for regression (Hyndman &amp; Koehler, [<reflink idref="bib34" id="ref72">34</reflink>]; Miles, [<reflink idref="bib51" id="ref73">51</reflink>]), and Accuracy, Precision, Recall, and F1-score for classification (Powers, [<reflink idref="bib61" id="ref74">61</reflink>]).</p> <p>One primary concern during evaluation is to check the model's <emph>generalizability</emph>. Specifically, <emph>overfitting</emph> refers to a model that performs exceptionally well on training data, but poorly on unseen data. Conversely, <emph>underfitting</emph> occurs when the model is too simple to capture underlying data patterns, resulting in suboptimal performance on both training and testing data (Goodfellow et al., [<reflink idref="bib27" id="ref75">27</reflink>]). In our analyses using the ACTIVE data set, overfitting can occur if we develop an overly complex neural network model. In this case, the model may have almost perfect predictive accuracy on the training data, suggesting that it essentially memorizes the data set, including the noise and outliers, rather than learning the underlying patterns. However, when this model is tested on a test set, its accuracy can drop dramatically because its learned patterns do not generalize well beyond the training data. On the other hand, an underfitted model may be due to the use of overly simple methods, such as linear regression that predicts the same scores using only one or two predictors (e.g., age and sex). Such a model may perform equally poorly on both the training and test sets because it is too simple to capture the complex relationship between input and outcome variables.</p> <hd id="AN0193318148-17">Software tools for neural network modeling</hd> <p>Several widely used software packages in both Python and R can facilitate the construction and training of neural networks. In Python, the most popular libraries include PyTorch (Paszke et al., [<reflink idref="bib57" id="ref76">57</reflink>]) and TensorFlow (Abadi et al., [<reflink idref="bib49" id="ref77">49</reflink>]), both of which offer flexible and powerful tools for defining, training, and evaluating neural networks. For those seeking higher-level APIs, Keras (Chollet et al., [<reflink idref="bib15" id="ref78">15</reflink>]), which runs on top of TensorFlow, provides a user-friendly interface for building models with minimal code. In this paper, we utilized PyTorch to construct the model and conduct all experiments.</p> <p>In R, the torch (Falbel &amp; Luraschi, [<reflink idref="bib22" id="ref79">22</reflink>]), tensorflow (Allaire &amp; Tang, [<reflink idref="bib5" id="ref80">5</reflink>]), and Keras (Chollet et al., [<reflink idref="bib16" id="ref81">16</reflink>]) packages provide R interfaces to the corresponding Python libraries, allowing users to build deep learning models in R with similar functionality. Other R-based packages include nnet (Venables &amp; Ripley, [<reflink idref="bib81" id="ref82">81</reflink>]) for simple feed-forward neural networks and neuralnet (Fritsch et al., [<reflink idref="bib23" id="ref83">23</reflink>]) for more customizable architectures, though these are more limited in scalability.</p> <p>Researchers can choose the programming environment based on their familiarity and project needs. Python is typically favored for large-scale or production-level applications, while R is often chosen for integration with statistical workflows and data analysis pipelines.</p> <hd id="AN0193318148-18">Optimizing model performance</hd> <p>The performance of a neural network is associated with a variety of elements. Optimizing these elements can be crucial for maximizing prediction accuracy. Key factors include hyperparameters like the learning rate of SGD and the size of training batches, as well as architectural aspects such as the number of neurons and layers. This section is designed to demonstrate the impact of each factor on the model's performance, offering practical insights for researchers looking to enhance their neural network models.</p> <p>We base our exploration on the FNN illustrated in Figure 2. This model, with a single hidden layer of 4 neurons, was initially trained with an SGD learning rate of 0.0001 and a batch size of 64. In the following subsections, we will methodically adjust several elements—learning rate, batch size, number of neurons, and number of layers—and examine their influence on the model's performance. At the end, we introduce grid search, an approach for identifying the most suitable hyperparameters and network architecture for a given data set.</p> <hd id="AN0193318148-19">Adjusting learning rate</hd> <p>The learning rate in SGD controls the step size of parameter updates toward minimizing the loss function. A learning rate that is too small can lead to premature convergence to the local minimum. Conversely, a too big learning rate might cause the model to oscillate without stabilizing, prolonging the convergence time and also increasing the chance of reaching suboptimal solutions. The optimal selection of learning rates is crucial for efficient and effective model training.</p> <p>In our experiment, we varied the learning rate across a spectrum from 0.1 to 0.00001. To ensure the reliability of the result, we trained 100 distinct FNN models for each learning rate value (i.e., 100 replications under each condition) using the same training set (To reduce randomness). Their performance was evaluated on the validation set, applying an early stopping strategy to halt training if the validation loss decreased by 0.0001 or less over 20 consecutive epochs. Note that some models did not converge due to issues such as vanishing or exploding gradients. We remove these models and will discuss this matter in later subsections.</p> <p>The compiled results are presented in Figure 3, showcasing a box plot that depicts the distribution of MSE across converged replications at each learning rate setting. We noted an initial reduction in MSE as the learning rate increased from 0.00001 to 0.001. However, a subsequent rise in the mean and variance of MSE was observed when the learning rate was further increased from 0.01 to 0.1, aligned with the mechanism described above.</p> <p>PHOTO (COLOR): Figure 3. Model performance with different SGD learning rates.</p> <hd id="AN0193318148-20">Changing batch size</hd> <p>The value of batch size is another hyperparameter in neural network training, which profoundly impacts model performance. Larger batch sizes typically offer a more precise gradient estimation, promoting stable and consistent updates during gradient descent. However, they would require more computing resources and carry the risk of converging to local minima, potentially inhibiting the optimal performance.</p> <p>In contrast, smaller batch sizes introduce a level of noise that acts as a regularizer (Wilson &amp; Martinez, [<reflink idref="bib82" id="ref84">82</reflink>]), often resulting in a reduced generalization error and a faster and more robust convergence path (Masters &amp; Luschi, [<reflink idref="bib50" id="ref85">50</reflink>]). Smaller batch sizes also align well with environments with memory constraints, such as GPU-based training. However, training with smaller batches typically requires a greater number of training steps and may also require a lower learning rate to preserve stability, given the higher variance in gradient estimation. These factors can potentially increase the total run time of the training process (Goodfellow et al., [<reflink idref="bib27" id="ref86">27</reflink>]).</p> <p>In practical applications, smaller batch sizes are generally favored. Bengio ([<reflink idref="bib7" id="ref87">7</reflink>]) recommended a batch size range from 1 to a few hundred for effective training, with 32 as a common default. Research by Masters and Luschi ([<reflink idref="bib50" id="ref88">50</reflink>]) indicates optimal test performance even with batch sizes as small as 2. Nonetheless, the ideal batch size is contingent on factors like the neural network's architecture and the data set's characteristics (Radiuk, [<reflink idref="bib64" id="ref89">64</reflink>]).</p> <p>Batch sizes are commonly set to powers of two (e.g., 8, 16, 32, 64, 128) primarily due to the computational efficiencies. Modern GPUs and CPUs handle operations more efficiently when working with sizes aligned to powers of two, optimizing parallel processing capabilities and memory management. When designing deep learning experiments, researchers typically set memory-related hyperparameters, such as batch sizes and the number of neurons per layer in the neural network, to powers of two for efficiency reasons and to simplify systematic exploration of the parameter space (Masters and Luschi ([<reflink idref="bib50" id="ref90">50</reflink>]; Keskar et al. ([<reflink idref="bib39" id="ref91">39</reflink>]).</p> <p>The base FNN model depicted in Figure 2 used a batch size of 64. To explore the impact of different batch sizes, we conducted experiments with batch sizes of 8, 16, 32, 64, and 128, while keeping other parameters constant. Consistent with our approach for learning rate adjustment, each batch size setting underwent 100 replications. The outcomes on the validation set are illustrated in Figure 4. Although the average performance differences across various batch sizes were subtle, we noted that larger batch sizes led to a significantly higher variance in MSE, suggesting decreased consistency in model predictions.</p> <p>PHOTO (COLOR): Figure 4. Model performance with different training batch sizes.</p> <hd id="AN0193318148-21">Adding more neurons</hd> <p>The complexity of a neural network, influenced by its number of neurons and layers, significantly affects its ability to model complex functions. An increase in the number of neurons enhances the network's capacity to learn intricate patterns, aiding in more accurate function approximation. However, an excess of neurons can lead to overfitting, where the model excessively learns the idiosyncrasies of the training data, including noise and anomalies, resulting in poor performance on new, unseen data. Moreover, a larger number of neurons increase the model's computational complexity, extending training times and demanding more computational resources. The ideal number of neurons should be determined considering the task complexity, available data volume, and the desired equilibrium between model generalization and specificity (Qiao et al., [<reflink idref="bib62" id="ref92">62</reflink>]). Tasks involving more complex relationships or higher-dimensional input data may benefit from more hidden layers, while simpler tasks or smaller data sets may favor smaller architectures to mitigate overfitting. Increasing the number of neurons expands the model's capacity to approximate complex, nonlinear functions, which enhances specificity but can reduce generalizability if not matched to the complexity of the data. In practice, researchers often begin with a reasonable range of neuron counts and empirically evaluate performance using systematic methods, as we further explain in Section 5.5.</p> <p>We start by investigating a single-hidden-layer FNN model with different numbers of neurons. Specifically, we increased the number of neurons from 4 to 8, 32, 64, and 128. As shown in Figure 5, as the neuron count increases, there is a continuous increase in MSE on the validation set. This suggests that for simpler data sets, a single-layer FNN with a lower neuron count may be adequate.</p> <p>PHOTO (COLOR): Figure 5. Model performance with different neuron counts in the hidden layer.</p> <hd id="AN0193318148-22">Adding more layers</hd> <p>Our investigation extended to the influence of augmenting the number of hidden layers in the FNN. We tested configurations comprising 1, 2, and 3 hidden layers, each with 4, 32, and 128 neurons. In each model, all hidden layers contained an equal number of neurons. For instance, in the 4-neuron scenario, the configurations included a single-layer model with four neurons, a two-layer model with</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn&gt;4&lt;/mn&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mn&gt;4&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> neurons, and a three-layer model with</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn&gt;4&lt;/mn&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mn&gt;4&lt;/mn&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mn&gt;4&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> neurons. This exploration led to a total of nine distinct conditions.</p> <p>The outcomes are illustrated in Figure 7. In scenarios with four neurons per layer, the MSE exhibits minimal variation across the different layer counts. This suggests that even a single-layer FNN is sufficiently complex for capturing the underlying patterns in the data. A similar trend is observed in the 32-neuron per-layer setups. With 128 neurons per layer, an increase in the number of layers seems to correlate with a decrease in MSE, indicating enhanced model performance. However, the configuration with 3 layers and 128 neurons per layer shows comparable performance with models having 2 layers and 4 neurons per layer. These observations suggest that increasing network depth is not necessarily associated with the improved predictive accuracy.</p> <hd id="AN0193318148-23">Hyperparameter optimization</hd> <p>Having explored various hyperparameters and their impact on model performance, the question arises: How can we identify an effective combination of these parameters? One common technique is <emph>grid search</emph>, which employs an exhaustive search strategy to explore a predefined space of hyperparameter combinations by constructing a grid (Hutter et al., [<reflink idref="bib33" id="ref93">33</reflink>]). Each point on the grid represents a unique set of hyperparameters. By evaluating the model's performance for each combination, grid search identifies the optimal set that yields the best performance according to a predefined metric, such as MSE. This approach ensures that all possible combinations within the specified range are considered, providing a comprehensive picture of how different hyperparameters affect a model's performance. Grid search is particularly suitable for the psychological research in which the sample size is often not too big.</p> <p>When optimizing the FNN using our example data, we implemented grid search by iterating over various learning rates (0.1, 0.01, 0.001, 0.0001, and 0.00001), batch sizes (<reflink idref="bib8" id="ref94">8</reflink>, 16, 32, 64, and 128), and numbers of neurons (<reflink idref="bib4" id="ref95">4</reflink>, 8, 32, 64, and 128) and layers (<reflink idref="bib1" id="ref96">1</reflink>, 2, and 3). Table 2 shows the top 10 models sorted by the lowest mean MSE. The best model configuration simply based on MSE consists of 2 layers with 64 neurons each, a learning rate of 0.0001, and a batch size of 8. This model converged in 89 out of 100 replications. Among these successful cases, the mean MSE was 30.597, and the average number of training epochs required for convergence was 93. However, since the performance differences among the top models in the table are marginal, the final selection depends on researcher preference, which is often guided by principles like <emph>model parsimony</emph>. This criterion, commonly valued in the social sciences, favors the simplest model architecture when predictive accuracy is comparable across more complex alternatives.</p> <p>Table 2. Top 10 models from grid search (with 3 predictors). CR: convergence rate. MSE and the number of training epochs are based on converged cases.</p> <p> <ephtml> &lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;ID&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;# layers&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;# neurons&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Learning rate.&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Batch size.&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;CR (%)&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;MSE&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;# training epochs&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;mean&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;std.&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;mean&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;std.&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody valign="top"&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;2&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;64&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.0001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;89&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.597&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.983&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;93.011&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;21.506&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;2&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;3&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;32&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;64&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;71&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.666&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;5.741&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;49.648&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;14.344&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;3&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;32&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;83&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.700&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1.771&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;44.940&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;15.607&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;3&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;64&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.0001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;16&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;74&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.742&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.789&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;84.622&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;19.176&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;5&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;32&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;64&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;93&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.751&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;5.526&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;49.441&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;16.907&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;6&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;3&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;64&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.0001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;70&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.762&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1.611&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;71.614&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;21.716&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;7&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;2&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;128&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.00001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;93&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.764&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1.087&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;394.108&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;78.920&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;3&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;64&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;57&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.769&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4.532&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;52.386&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;29.186&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;9&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;2&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;128&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.0001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;93&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.782&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;2.362&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;84.538&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;20.004&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;10&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;3&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;64&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;128&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.790&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;&amp;#8211;&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;26.000&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;&amp;#8211;&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>Besides prediction accuracy and training efficiency, researchers should also pay attention to the convergence issue when selecting models. For instance, the 10th-ranked model had a convergence rate of only 1%; that is, it converged once in 100 replications. Despite its high accuracy in this one case, such a low convergence rate undermines the model's reliability and generalizability and thus should not be selected. This example emphasizes the importance of evaluating models not only on their performance metrics, but also on their stability and consistency across multiple runs.</p> <p>Alternative optimization strategies are also widely used, especially when computational efficiency is a concern. For instantce, <emph>Random search</emph> (Bergstra &amp; Bengio, [<reflink idref="bib8" id="ref97">8</reflink>]) samples combinations randomly within the specified ranges and has been shown to outperform grid search in high-dimensional spaces. More advanced techniques such as <emph>Bayesian optimization</emph> (Snoek et al., [<reflink idref="bib74" id="ref98">74</reflink>]) model the performance landscape and guide the search toward promising regions based on prior evaluations. These methods can be particularly useful when training models is expensive or the hyperparameter space is large.</p> <p>In our case, grid search was implemented using base functions and custom scripts. For researchers seeking more scalable or automated solutions, several software packages are available. In R, tune (Kuhn, [<reflink idref="bib41" id="ref99">41</reflink>]) facilitates hyperparameter tuning for the tidymodels packages. mlr3tuning (Becker et al., [<reflink idref="bib6" id="ref100">6</reflink>]) finds optimal hyperparameter configurations for any 'mlr3′ learner with several optimization algorithms, including grid search, random search, and Bayesian optimization. In Python, libraries such as scikit-learn (Pedregosa et al., [<reflink idref="bib58" id="ref101">58</reflink>]), Optuna (Akiba et al., [<reflink idref="bib4" id="ref102">4</reflink>]), Hyperopt (Bergstra et al., [<reflink idref="bib9" id="ref103">9</reflink>]), and Ray Tune (Liaw et al., [<reflink idref="bib45" id="ref104">45</reflink>]) support a variety of tuning strategies.</p> <hd id="AN0193318148-24">Convergence rate</hd> <p>We further examine the effect of hyperparameters on model convergence. Figure 6 illustrates a case of non-convergence during training, where the training loss stagnated at an early stage, and the validation loss exhibited substantial fluctuations, suggesting being trapped in a local minimum. The convergence rates for different hyperparameter configurations are shown in Figure 8. Specifically, Figure 8(a) shows a clear trend where the convergence rate initially increases with the learning rate, but decreases as the learning rate continues to increase. This pattern is similar to the association between learning rate and MSE. On the other hand, as shown in Figure 8(b), the effect of increasing the batch size seems minimal and generally results in a slight decrease in the probability of convergence. Meanwhile, Figure 8(c) shows that adding more neurons per layer tends to help with model convergence in general; however, all three plots suggest that adding more layers tends to decrease the likelihood of model convergence.</p> <p>PHOTO (COLOR): Figure 6. Example of non-convergence.</p> <p>PHOTO (COLOR): Figure 7. Model performance with different numbers of layers.</p> <p>PHOTO (COLOR): Figure 8. Model convergence rate across different hyperparameter settings.</p> <hd id="AN0193318148-25">Other influencing factors</hd> <p>In addition to the hyperparameters that are directly related to a FNN model, there are other factors that can influence the performance of a model.</p> <hd id="AN0193318148-26">Number of predictors</hd> <p>Incorporating more relevant predictors will generally improve the model's performance. To illustrate this effect, we extended our initial 3 predictors (age, education, and sex) to 10 predictors: age, education, sex, site, booster, ufov, mmse, hvltt, hvltt2, and hvltt3. The performance outcomes are shown in Table 3. The optimal model converged in 90% cases with a mean MSE of 17.430. When compared to the results presented in Table 2, it is evident that adding these predictors substantially reduced the MSE and increased the convergence rate.</p> <p>Table 3. Top 10 models from grid search (with 10 predictors). CR: convergence rate. MSE and the number of training epochs are based on converged cases.</p> <p> <ephtml> &lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;ID&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;# layers&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;# neurons&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Learning rate.&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;Batch size.&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;CR (%)&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;MSE&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;# training epochs&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;mean&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;std.&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;mean&lt;/p&gt;&lt;/td&gt;&lt;td&gt;&lt;p&gt;std.&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody valign="top"&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.0001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;90&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.430&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1.023&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;92.378&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;28.080&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;2&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;16&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;88&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.447&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.710&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;46.466&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;22.773&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;3&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.00001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;82&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.464&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1.692&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;414.354&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;154.924&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;64&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;91&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.525&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;2.956&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;56.198&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;16.578&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;5&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.0001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;16&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;90&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.527&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.650&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;108.289&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;30.446&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;6&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.0001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;93&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.599&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.918&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;103.387&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;28.296&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;7&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;128&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;86&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.638&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.795&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;73.849&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;18.637&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;8&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;85&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.656&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.696&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;35.894&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;11.484&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;9&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.0001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;32&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;79&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.661&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1.344&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;110.190&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;25.181&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;&lt;p&gt;10&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;4&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;0.001&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;32&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;85&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;17.689&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;1.652&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;52.894&lt;/p&gt;&lt;/td&gt;&lt;td char="."&gt;&lt;p&gt;15.217&lt;/p&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt; </ephtml> </p> <p>Interestingly, we also observed a shift toward simpler architectures: All of the 10 best-performing models in this extended setting consisted of only 1 hidden layer. This may indicate that the additional predictors made the underlying patterns easier to learn, reducing the need for deep intermediate representations. With richer input features, even shallow networks can capture the relevant relationships without relying on deeper architectures.</p> <p>Moreover, deeper networks can sometimes introduce unnecessary complexity, which may hinder optimization and increase the risk of overfitting—especially when the feature set is limited. In fact, when comparing Table 2 with Table 3, it appears that the previous models trained with only three predictors may have been overfitting to the limited information available.</p> <hd id="AN0193318148-27">Sample sizes</hd> <p>Number of samples in the data set can significantly affect the prediction results of a neural network. Larger sample sizes typically yield better performance by providing a more comprehensive representation of the data distribution, while smaller sizes may lead to outfitting due to insufficient data. To test this, we built seven data sets with varied sample sizes (<emph>N</emph> = 25, 50, 100, 200, 300, 500, 700, 1,000, 1250, and 1,500) by randomly drawing subsets of samples from the original data set. As we did before, each data set was then randomly divided into training/evaluation/test sets with a 72:13:15 ratio. For every distinct sample size, we trained 100 separate FNN models with the default configurations and evaluated their performance. Note that we did the 100 replications for each different sample size on a fixed subset of samples instead of 100 random subsets of samples. This was to avoid importing extra randomness from different data sets.</p> <p>The model was trained using the 10 predictors outlined previously. Figure 9(a) illustrates the variation in MSE across different sample sizes. With the smallest sample size (</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mi&gt;N&lt;/mi&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;25&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> ), the MSE exhibited a high mean and high variance, indicating that the data set was too limited for effective training. Increasing the sample size from 25 to 50 led to a notable decrease in MSE. Further increasing the sample size continued to lower the mean MSE and slightly reduced its variance, thereby enhancing the model's accuracy and consistency. Regarding the number of training epochs, as depicted in Figure 9(b), very small data sets required an exceedingly large number of epochs since the model struggled to learn from the scant data, resulting in instability. As the sample size grew, the required number of training epochs diminished, reflecting improved learning efficiency. However, the number of epochs rose again with very large data sets, probably attributed to the complexity of learning from a large pool of data.</p> <p>PHOTO (COLOR): Figure 9. Model performance with different sample sizes.</p> <hd id="AN0193318148-28">Training and validation set sizes</hd> <p>The way in which the data is divided into a training set and a validation set can also affect the performance of the model. Specifically, allocating too large a portion of the data set to the validation set can prevent the model from getting enough training data, which can lead to underfitting; that is, the model is unable to effectively capture the underlying patterns in the data. Conversely, allocating too small a portion to the validation set may cause large variation in the validation step, thus preventing accurate optimization judgments such as when to terminate training and which model to choose. Here, we include a set of experiments that vary the ratio of training and validation set sizes to illustrate these effects. This is not part of the core modeling procedure, but is presented as additional context to help researchers remain mindful of data set composition when interpreting validation performance.</p> <p>To empirically demonstrate these effects, we kept 15% of the original data as the test set in our experiments and varied the ratio of the training set to the validation set so that the validation set accounted for 10%, 20%, 30%, ..., 90% of the rest of the data set, respectively.</p> <p>Figure 10 shows the results. When the validation set comprises only 10% of the visible data set (i.e., the combination of training and validation set), it contains</p> <p>Graph</p> <p> <ephtml> &lt;math display="inline" xmlns="http://www.w3.org/1998/Math/MathML"&gt;&lt;mrow&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mo&gt;,&lt;/mo&gt;&lt;mn&gt;573&lt;/mn&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mn&gt;0.85&lt;/mn&gt;&lt;mo&gt;&amp;#215;&lt;/mo&gt;&lt;mn&gt;0.1&lt;/mn&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;133&lt;/mn&gt;&lt;/mrow&gt;&lt;/math&gt; </ephtml> participants, which is insufficient to accurately reflect the model's performance, resulting in inappropriate early stopping and a high MSE. As the proportion of the validation set increases to 0.4, the MSE decreases, but beyond that point, the MSE starts to increase again, which is mainly caused by insufficient training data. This suggests that for this particular data set and the FNN model, allocating 40% of the visible data to the validation set is likely the optimal choice. Notably, Figure 10(b) shows a positive correlation between validation set size and the number of epochs required for training. One plausible explanation is that a smaller training set contains less information from the predictors, which might cause underfitting and require more epochs to converge.</p> <p>PHOTO (COLOR): Figure 10. Model performance with different validation set sizes.</p> <hd id="AN0193318148-29">Model interpretation</hd> <p>Interpreting the outcomes of deep learning models is crucial yet difficult, particularly in fields like psychology and other social sciences, where understanding the model's predictions can offer valuable insights into human behaviors and decision-making processes (Hassija et al., [<reflink idref="bib29" id="ref105">29</reflink>]; Orrù et al., [<reflink idref="bib56" id="ref106">56</reflink>]; Ribeiro et al., [<reflink idref="bib65" id="ref107">65</reflink>]; Rudin, [<reflink idref="bib68" id="ref108">68</reflink>]). This section outlines two simple, accessible methods for interpreting model results.</p> <p>One common approach for model interpretation is partial dependence plots (PDPs), which visualizes the relationship between the target variable and a subset of input features of interest while marginalizing over the values of all other input features (Hastie et al., [<reflink idref="bib30" id="ref109">30</reflink>]; Pedregosa et al., [<reflink idref="bib58" id="ref110">58</reflink>]). Intuitively, partial dependence can be interpreted as the expected target response as a function of the selected input features. By plotting the partial dependence, one can gain insights into the marginal effect of the input features on the model's predictions. Unlike simple bivariate plots, which show raw relationships between two observed variables, PDPs depict the marginal effect of selected input features on the model's predictions while averaging out the influence of other variables in the model.</p> <p>Figure 11 shows three one-way PDPs, each plotting a single input variable against the predicted outcome. From Figure 11(a), we observe a trend where the model predicts better test performance with younger participants. Similarly, Figure 11(b) indicates that in general, participants with longer education years are predicted to perform better in tests. Additionally, Figure 11(c) reveals a tendency of the model to predict higher test scores for female participants. These plots facilitate a potential understanding of the relationships between different input variables and predicted outcomes.</p> <p>PHOTO (COLOR): Figure 11. One-way partial dependence plots (PDPs).</p> <p>In addition to visualization, there are various methods to assess feature importance in neural network models (Molnar, [<reflink idref="bib54" id="ref111">54</reflink>]). A straightforward technique is feature permutation, that is, randomly shuffling the values of each input variable and observing the impact on the network's performance (Breiman, [<reflink idref="bib12" id="ref112">12</reflink>]). A notable increase in MSE upon permuting a variable suggests its significant influence on the model's predictions. This process is repeated for each input variable (or subset of variables) to determine their relative importance.</p> <p>In our study using the ACTIVE data set with 10 variables, we observed distinct changes in model performance upon permutation. We conducted 500 replications, where in each replication, we randomly split the data into training, validation, and test sets. During the testing stage, we first tested the model on the original test set, then permuted each predictor in the test set and allowed the model to make predictions. As shown in Figure 12, permuting <emph>hvltt3</emph>, <emph>hvltt</emph>, and <emph>hvltt2</emph> led to the most substantial increase compared to the original average MSE (the red-dotted line), indicating that previous test scores could be the most substantial predictors for accurately predicting the latest test score. Permuting the demographic variables <emph>sex</emph> and <emph>age</emph> also led to a slight increase in MSE, indicating their importance. The remaining predictors did not result in significant differences, which may suggest either that they are correlated with other predictors and thus contribute to little additional information, or that they are irrelevant.</p> <p>PHOTO (COLOR): Figure 12. MSE with permuted predictors.</p> <p>Beyond PDPs and feature permutation, more advanced model interpretation techniques are available and widely used in the machine learning community. One such method is SHapley Additive exPlanations (SHAP), which provides consistent and locally accurate attributions for each feature's contribution to a model's prediction (Lundberg &amp; Lee, [<reflink idref="bib47" id="ref113">47</reflink>]). SHAP values can offer deeper insight into individual predictions and the global behavior of the model. We encourage readers to explore SHAP and other interpretability tools, such as LIME (Ribeiro et al., [<reflink idref="bib65" id="ref114">65</reflink>]) and Integrated Gradients (Sundararajan et al., [<reflink idref="bib76" id="ref115">76</reflink>]), especially when applying neural networks in high-stakes or complex research settings.</p> <hd id="AN0193318148-30">Discussion</hd> <p>In this article, we presented a step-by-step guide for employing FNNs in psychological research. We started with a concrete example of a basic 3-layer FNN on a psychological data set, including the matrix representation, the forward propagation, and the backpropagation process. Subsequent experiments demonstrated the impact of hyperparameter adjustments and network structure variations on model performance. Additionally, we showed two simple ways of interpreting neural network predictions and assessing feature importance.</p> <hd id="AN0193318148-31">FNNs as a foundation for deep learning</hd> <p>FNNs are the simplest form of neural networks. Subsequent innovation builds on its foundation but introduces new architecture and mechanisms to handle more complex data patterns, especially in sequence and spatial recognition tasks. These advancements allow for better performance in tasks involving sequences (like text and audio) and spatial data (like images). Specifically, Recurrent Neural Networks (RNNs, Rumelhart et al. ([<reflink idref="bib69" id="ref116">69</reflink>])) and Convolutional Neural Networks (CNNs, LeCun et al. ([<reflink idref="bib43" id="ref117">43</reflink>])) evolved from FNNs.</p> <p>RNNs can handle sequences by maintaining a memory of previous inputs, making them highly useful for analyzing psychological time-series data, such as predicting suicidal ideation in ecological momentary assessments (EMA; Choo ([<reflink idref="bib17" id="ref118">17</reflink>])). However, standard RNNs struggled with vanishing gradients, limiting their ability to model long-range dependencies. To overcome this, Long Short-Term Memory (LSTM) networks (Hochreiter &amp; Schmidhuber, [<reflink idref="bib32" id="ref119">32</reflink>]) introduced gating mechanisms to preserve context over extended sequences. Gated Recurrent Units (GRUs; Cho et al. ([<reflink idref="bib14" id="ref120">14</reflink>])) further streamlined this approach, offering efficient training on smaller data sets. These models have been applied in areas such as tracking child language development (Portelance et al., [<reflink idref="bib60" id="ref121">60</reflink>]; Sagae, [<reflink idref="bib71" id="ref122">71</reflink>]) and diagnosing health conditions using data from electroencephalogram (EEG; Cui et al. ([<reflink idref="bib20" id="ref123">20</reflink>])) and machine health monitoring systems (MHMS; Zhao et al. ([<reflink idref="bib86" id="ref124">86</reflink>])).</p> <p>CNNs excel in pattern recognition within image data through convolutional layers. Revolutionized spatial data analysis through convolutional layers detect hierarchical patterns in grid-like data, for example, images (LeCun et al., [<reflink idref="bib43" id="ref125">43</reflink>]). In psychology, CNNs have automated facial expression analysis, reducing reliance on subjective manual coding in emotion research (Corneanu et al., [<reflink idref="bib19" id="ref126">19</reflink>]). They have also applied in neuroimaging, such as predicting brain age from raw MRI data (Cole et al., [<reflink idref="bib18" id="ref127">18</reflink>]), identifying biomarkers for conditions traditionally diagnosed through behavioral observation.</p> <p>Transformers, introduced by Vaswani et al. ([<reflink idref="bib80" id="ref128">80</reflink>]), disrupted sequential processing paradigms by leveraging self-attention mechanisms to weigh interdependencies across entire data sets. Models like BERT (Devlin et al., [<reflink idref="bib21" id="ref129">21</reflink>]) and the GPT series (Brown et al., [<reflink idref="bib13" id="ref130">13</reflink>]; Radford et al., [<reflink idref="bib63" id="ref131">63</reflink>]) enabled breakthroughs in natural language processing (NLP). For psychologists, these tools facilitate sentiment analysis of free-text responses (Hoang et al., [<reflink idref="bib31" id="ref132">31</reflink>]), personality inference from social media (Peters &amp; Matz, [<reflink idref="bib59" id="ref133">59</reflink>]), and even automated therapeutic chatbots (Minerva &amp; Giubilini, [<reflink idref="bib53" id="ref134">53</reflink>]). Transformers also support multimodal approaches, such as predicting depressive symptoms from text, audio, video, and sensor data (Lorenzoni et al., [<reflink idref="bib46" id="ref135">46</reflink>]; Moura et al., [<reflink idref="bib55" id="ref136">55</reflink>]), demonstrating their versatility in capturing nuanced behavioral signals.</p> <p>Nevertheless, it is important to note that these advanced models still rest on the foundational principles of FNNs. The understanding of core concepts, such as backpropagation, loss functions, and the intricacies of training and testing, is universal across these neural network architectures. The hyperparameter tuning strategies for FNNs are also applicable to more advanced models.</p> <hd id="AN0193318148-32">Comparison with traditional statistical models</hd> <p>FNNs and other machine learning models can offer meaningful advantages over traditional statistical models, such as linear regression, logistic regression, multilevel modeling, and structural equation modeling (SEM), under the certain conditions. Traditional modeling techniques are often theory-driven, interpretable, and effective at hypothesis testing and understanding causal relationships. However, they rely on explicit assumptions (e.g., linearity, homoscedasticity, normality) and predefined model structures, potentially limiting their flexibility when analyzing complex behavioral data.In contrast, neural networks provide greater flexibility and predictive accuracy by automatically detecting complex, nonlinear interactions and patterns without explicit specification. This feature is particularly advantageous for high-dimensional data sets or when dealing with subtle, intricate relationships among variables.</p> <p>Importantly, machine learning methods and traditional modeling approaches should not be viewed as competing methodologies but rather complementary tools. Integrating nerual network models into psychological research allows researchers to capitalize on their strengths in exploratory analysis and predictive performance, guiding subsequent confirmatory analyses and theory-building with traditional statistical techniques.</p> <hd id="AN0193318148-33">Loss functions and learning paradigms</hd> <p>Although not considered in this tutorial, the type of loss functions, depending on the task type, also play an important role. Our primary task in this article was regression, and we employed MSE as the loss function. However, when dealing with categorical data, other loss functions are more appropriate. For example, in classification tasks, cross-entropy loss is commonly used (Goodfellow et al., [<reflink idref="bib27" id="ref137">27</reflink>]; Hastie et al., [<reflink idref="bib30" id="ref138">30</reflink>]). In binary classification scenarios, binary cross-entropy is effective (Ruby &amp; Yendapalli, [<reflink idref="bib67" id="ref139">67</reflink>]), while for multi-class problems, categorical cross-entropy and its variations are more suitable (Gordon-Rodriguez et al., [<reflink idref="bib28" id="ref140">28</reflink>]; Rusiecki, [<reflink idref="bib70" id="ref141">70</reflink>]). These loss functions are better aligned with the nature of categorical data and help accurately model the probability distribution of the class labels.</p> <p>Finally, our experiments employed <emph>supervised learning</emph> (Hastie et al., [<reflink idref="bib30" id="ref142">30</reflink>]), as we had labeled training data, that is, the actual values of the outcome variables. However, there are research scenarios where <emph>unsupervised learning</emph> could be beneficial (Xu &amp; Wunsch, [<reflink idref="bib83" id="ref143">83</reflink>]). For instance, tasks like topic modeling (Blei et al., [<reflink idref="bib11" id="ref144">11</reflink>]) or clustering (Jain, [<reflink idref="bib36" id="ref145">36</reflink>]), which aim to discover inherent patterns or groupings in the data without pre-defined labels, can be approached with unsupervised learning techniques (Ghahramani, [<reflink idref="bib25" id="ref146">25</reflink>]; Suominen &amp; Toivanen, [<reflink idref="bib77" id="ref147">77</reflink>]).</p> <hd id="AN0193318148-34">Article information</hd> <p> <bold>Conflict of interest disclosures:</bold> Each author signed a form for disclosure of potential conflicts of interest. No authors reported any financial or other conflicts of interest in relation to the work described.</p> <p> <bold>Ethical Principles:</bold> The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data.</p> <p> <bold>Funding:</bold> This work was supported by Grant R305D210023 from the Institute of Education Sciences, the Lucy Family Institute for Data and Society, and Notre Dame Global.</p> <p> <bold>Role of the Funders/Sponsors:</bold> None of the funders or sponsors of this research had any role in the design and conduct of the study; collection, management, analysis, and interpretation of data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.</p> <p> <bold>Acknowledgments:</bold> The authors would like to thank Hanna Weber for their comments on prior versions of this manuscript. The ideas and opinions expressed herein are those of the authors alone, and endorsement by the authors' institution or the funding agencies is not intended and should not be inferred.</p> <ref id="AN0193318148-35"> <title> References </title> <blist> <bibl id="bib1" idref="ref14" type="bt">1</bibl> <bibtext> Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467</bibtext> </blist> <blist> <bibl id="bib2" idref="ref13" type="bt">2</bibl> <bibtext> Abdurahman, S., Vu, H., Zou, W., Ungar, L., &amp; Bhatia, S. (2024). A deep learning approach to personality assessment: Generalizing across items and expanding the reach of survey-based research. Journal of Personality and Social Psychology, 126 (2), 312 – 331. https://doi.org/10.1037/pspp0000480</bibtext> </blist> <blist> <bibl id="bib3" idref="ref28" type="bt">3</bibl> <bibtext> Agarap, A. F. (2018). Deep learning using rectified linear units (relu). arXiv Preprint, arXiv:1803.08375</bibtext> </blist> <blist> <bibl id="bib4" idref="ref29" type="bt">4</bibl> <bibtext> Akiba, T., Sano, S., Yanase, T., Ohta, T., Koyama, M. (2019). Optuna: A next-generation hyperparameter optimization framework. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining, 2623 – 2631.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref30" type="bt">5</bibl> <bibtext> Allaire, J., Tang, Y. (2025). Tensorflow: R interface to 'tensorflow' [R package version 2.16.0.9000]. https://github.com/rstudio/tensorflow</bibtext> </blist> <blist> <bibl id="bib6" idref="ref31" type="bt">6</bibl> <bibtext> Becker, M., Lang, M., Richter, J., Bischl, B., Schalk, D. (2025). Mlr3tuning: Hyperparameter optimization for 'mlr3′ [R package version 1.4.0]. https://github.com/mlr-org/mlr3tuning</bibtext> </blist> <blist> <bibl id="bib7" idref="ref35" type="bt">7</bibl> <bibtext> Bengio, Y. (2012). Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trade (2nd ed., pp. 437 – 478). Springer.</bibtext> </blist> <blist> <bibl id="bib8" idref="ref36" type="bt">8</bibl> <bibtext> Bergstra, J., &amp; Bengio, Y. (2012). Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13 (1), 281 – 305.</bibtext> </blist> <blist> <bibl id="bib9" idref="ref37" type="bt">9</bibl> <bibtext> Bergstra, J., Komer, B., Eliasmith, C., Yamins, D., &amp; Cox, D. D. (2015). Hyperopt: A python library for model selection and hyperparameter optimization. Computational Science &amp; Discovery, 8 (1), 014008. https://doi.org/10.1088/1749-4699/8/1/014008</bibtext> </blist> <blist> <bibtext> Bishop, C. M., &amp; Nasrabadi, N. M. (2006). Pattern recognition and machine learning (Vol. 4) Springer.</bibtext> </blist> <blist> <bibtext> Blei, D. M., Ng, A. Y., &amp; Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3 (Jan), 993 – 1022.</bibtext> </blist> <blist> <bibtext> Breiman, L. (2001). Random forests. Machine Learning, 45 (1), 5 – 32. https://doi.org/10.1023/A:1010933404324</bibtext> </blist> <blist> <bibtext> Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learn- ers. Advances in Neural Information Processing Systems, 33, 1877 – 1901.</bibtext> </blist> <blist> <bibtext> Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., &amp; Bengio, Y. (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv Preprint, arXiv:1406.1078.</bibtext> </blist> <blist> <bibtext> Chollet, F., et al. (2015). Keras [Computer software]. https://keras.io</bibtext> </blist> <blist> <bibtext> Chollet, F., Allaire, J., et al. (2017). R interface to keras [R package version 1.4.0]. https://github.com/rstudio/keras</bibtext> </blist> <blist> <bibtext> Choo, T.-H. J. (2025). Leveraging recurrent neural networks for predicting suicidal ideation: Advancing the analysis of ecological momentary assessment data. Columbia University.</bibtext> </blist> <blist> <bibtext> Cole, J. H., Poudel, R. P., Tsagkrasoulis, D., Caan, M. W., Steves, C., Spector, T. D., &amp; Montana, G. (2017). Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker. NeuroImage, 163, 115 – 124. https://doi.org/10.1016/j.neuroimage.2017.07.059</bibtext> </blist> <blist> <bibtext> Corneanu, C. A., Simón, M. O., Cohn, J. F., &amp; Guerrero, S. E. (2016). Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: History, trends, and affect-related applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38 (8), 1548 – 1568. https://doi.org/10.1109/TPAMI.2016.2515606</bibtext> </blist> <blist> <bibtext> Cui, R., Liu, M., &amp; Initiative, A. D. N., Alzheimer's Disease Neuroimaging Initiative (2019). Rnn-based longitudinal analysis for diagnosis of alzheimer's disease. Computerized Medical Imaging and Graphics: The Official Journal of the Computerized Medical Imaging Society, 73, 1 – 10. https://doi.org/10.1016/j.compmedimag.2019.01.005</bibtext> </blist> <blist> <bibtext> Devlin, J., Chang, M.-W., Lee, K., Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: human Language Technologies, volume 1 (Long and short papers), 4171 – 4186.</bibtext> </blist> <blist> <bibtext> Falbel, D., Luraschi, J. (2025). Torch: Tensors and neural networks with 'gpu' acceleration [R package version 0.16.0]. https://torch.mlverse.org/docs</bibtext> </blist> <blist> <bibtext> Fritsch, S., Guenther, F., Wright, M. N. (2020). Neuralnet: Training of neural networks [R package version 1.44.6]. https://github.com/bips-hb/neuralnet</bibtext> </blist> <blist> <bibtext> Gareth, J., Daniela, W., Trevor, H., &amp; Robert, T. (2013). An introduction to statistical learning: With applications in r. Spinger.</bibtext> </blist> <blist> <bibtext> Ghahramani, Z. (2003). Unsupervised learning. In Summer school on machine learning (pp. 72 – 112). Springer.</bibtext> </blist> <blist> <bibtext> Goldberg, Y. (2022). Neural network methods for natural language processing. Springer Nature.</bibtext> </blist> <blist> <bibtext> Goodfellow, I., Bengio, Y., &amp; Courville, A. (2016). Deep learning. MIT press.</bibtext> </blist> <blist> <bibtext> Gordon-Rodriguez, E., Loaiza-Ganem, G., Pleiss, G., &amp; Cunningham, J. P. (2020). Uses and abuses of the cross-entropy loss: Case studies in modern deep learning.</bibtext> </blist> <blist> <bibtext> Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., &amp; Hussain, A. (2024). Interpreting black-box models: A review on explainable artificial intelligence. Cognitive Computation, 16 (1), 45 – 74. https://doi.org/10.1007/s12559-023-10179-8</bibtext> </blist> <blist> <bibtext> Hastie, T., Tibshirani, R., Friedman, J. H., &amp; Friedman, J. H. (2009). The elements of statistical learning: Data mining, inference, and prediction. (Vol. 2) Springer.</bibtext> </blist> <blist> <bibtext> Hoang, M., Bihorac, O. A., Rouces, J. (2019). Aspect-based sentiment analysis using bert. Proceedings of the 22nd Nordic Conference on Computational Linguistics, 187 – 196.</bibtext> </blist> <blist> <bibtext> Hochreiter, S., &amp; Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9 (8), 1735 – 1780. https://doi.org/10.1162/neco.1997.9.8.1735</bibtext> </blist> <blist> <bibtext> Hutter, F., Kotthoff, L., &amp; Vanschoren, J. (2019). Automated machine learning: Methods, systems, challenges. Springer Nature.</bibtext> </blist> <blist> <bibtext> Hyndman, R. J., &amp; Koehler, A. B. (2006). Another look at measures of forecast accuracy. International Journal of Forecasting, 22 (4), 679 – 688. https://doi.org/10.1016/j.ijforecast.2006.03.001</bibtext> </blist> <blist> <bibtext> Iyortsuun, N. K., Kim, S.-H., Jhon, M., Yang, H.-J., &amp; Pant, S. (2023). A review of machine learning and deep learning approaches on mental health diagnosis. Healthcare, 11 (3), 285. https://doi.org/10.3390/healthcare11030285</bibtext> </blist> <blist> <bibtext> Jain, A. K. (2010). Data clustering: 50 years beyond k-means. Pattern Recognition Letters, 31 (8), 651 – 666. https://doi.org/10.1016/j.patrec.2009.09.011</bibtext> </blist> <blist> <bibtext> Jobe, J. B., Smith, D. M., Ball, K., Tennstedt, S. L., Marsiske, M., Willis, S. L., Rebok, G. W., Morris, J. N., Helmers, K. F., Leveck, M. D., &amp; Kleinman, K.,. (2001). Active: A cognitive intervention trial to promote independence in older adults. Controlled Clinical Trials, 22 (4), 453 – 479. https://doi.org/10.1016/s0197-2456(01)00139-8</bibtext> </blist> <blist> <bibtext> Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R., &amp; McHardy, R. (2023). Challenges and applications of large language models. arXiv Preprint, arXiv:2307.10169</bibtext> </blist> <blist> <bibtext> Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., &amp; Tang, P. T. P. (2016). On large-batch training for deep learning: Generalization gap and sharp minima. arXiv Preprint, arXiv:1609.04836</bibtext> </blist> <blist> <bibtext> Khan, S., Rahmani, H., Shah, S. A. A., Bennamoun, M., Medioni, G., &amp; Dickinson, S. (2018). A guide to convolutional neural networks for computer vision. (Vol. 8) Springer.</bibtext> </blist> <blist> <bibtext> Kuhn, M. (2025). Tune: Tidy tuning tools [R package version 2.0.0.9000]. https://tune.tidymodels.org/tune</bibtext> </blist> <blist> <bibtext> LeCun, Y., Bengio, Y., &amp; Hinton, G. (2015). Deep learning. Nature, 521 (7553), 436 – 444. https://doi.org/10.1038/nature14539</bibtext> </blist> <blist> <bibtext> LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., &amp; Jackel, L. D. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1 (4), 541 – 551. https://doi.org/10.1162/neco.1989.1.4.541</bibtext> </blist> <blist> <bibtext> LeCun, Y., Bottou, L., Orr, G. B., &amp; Müller, K.-R. (2002). Efficient backprop. In Neural networks: Tricks of the trade. (pp. 9 – 50) Springer.</bibtext> </blist> <blist> <bibtext> Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J. E., &amp; Stoica, I. (2018). Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118</bibtext> </blist> <blist> <bibtext> Lorenzoni, G., Velmovitsky, P. E., Alencar, P., Cowan, D. (2024). Gpt-4 on clinic depression assessment: An llm-based pilot study. 2024 IEEE International Conference on Big Data (BigData), 5043 – 5049.</bibtext> </blist> <blist> <bibtext> Lundberg, S. M., &amp; Lee, S.-I. (2017). A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 4768 – 4777.</bibtext> </blist> <blist> <bibtext> Malhotra, A., &amp; Jindal, R. (2022). Deep learning techniques for suicide and depression detection from online social media: A scoping review. Applied Soft Computing, 130, 109713. https://doi.org/10.1016/j.asoc.2022.109713</bibtext> </blist> <blist> <bibtext> M., Abadi, A., Agarwal, P., Barham, E., Brevdo, Z., Chen, C., Citro, G. S., Corrado, A., Davis, J., Dean, M., Devin, S., Ghemawat, I., Goodfellow, A., Harp, G., Irving, M., Isard, Jia, Y., Rafal Jozefowicz, L., Kaiser, M., Kudlur, X., Zheng. (2015). TensorFlow: Large-scale machine learning on heterogeneous systems [Software available from tensorflow.org]. https://<ulink href="http://www.tensorflow.org/">www.tensorflow.org/</ulink></bibtext> </blist> <blist> <bibtext> Masters, D., &amp; Luschi, C. (2018). Revisiting small batch training for deep neural networks. arXiv preprint arXiv:1804.07612</bibtext> </blist> <blist> <bibtext> Miles, J. (2005). R-squared, adjusted r-squared. Encyclopedia of statistics in behavioral science.</bibtext> </blist> <blist> <bibtext> Min, S., Lee, B., &amp; Yoon, S. (2017). Deep learning in bioinformatics. Briefings in Bioinformatics, 18 (5), 851 – 869. https://doi.org/10.1093/bib/bbw068</bibtext> </blist> <blist> <bibtext> Minerva, F., &amp; Giubilini, A. (2023). Is ai the future of mental healthcare? Topoi, 42 (3), 809 – 817. https://doi.org/10.1007/s11245-023-09932-3</bibtext> </blist> <blist> <bibtext> Molnar, C. (2020). Interpretable machine learning (3rd). Christoph Molnar. https://christophm.github.io/interpretable-ml-book/</bibtext> </blist> <blist> <bibtext> Moura, I., Teles, A., Viana, D., Marques, J., Coutinho, L., &amp; Silva, F. (2023). Digital phenotyping of mental health using multimodal sensing of multiple situations of interest: A systematic literature review. Journal of Biomedical Informatics, 138, 104278. https://doi.org/10.1016/j.jbi.2022.104278</bibtext> </blist> <blist> <bibtext> Orrù, G., Monaro, M., Conversano, C., Gemignani, A., &amp; Sartori, G. (2020). Machine learning in psychometrics and psychological research. Frontiers in Psychology, 10, 492685. https://doi.org/10.3389/fpsyg.2019.02970</bibtext> </blist> <blist> <bibtext> Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 4768 – 4777.</bibtext> </blist> <blist> <bibtext> Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., &amp; Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825 – 2830.</bibtext> </blist> <blist> <bibtext> Peters, H., &amp; Matz, S. C. (2024). Large language models can infer psychological dispositions of social media users. PNAS Nexus, 3 (6), pgae231. https://doi.org/10.1093/pnasnexus/pgae231</bibtext> </blist> <blist> <bibtext> Portelance, E., Degen, J., Frank, M. C. (2020). Predicting age of acquisition in early word learning using recurrent neuralnetworks. Proceedings of the Annual Meeting of the Cognitive Science Society, 42.</bibtext> </blist> <blist> <bibtext> Powers, D. M. (2020). Evaluation: From precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv Preprint, arXiv:2010.16061</bibtext> </blist> <blist> <bibtext> Qiao, J., Li, S., Han, H., &amp; Wang, D. (2017). An improved algorithm for building self-organizing feedforward neural networks. Neurocomputing, 262, 28 – 40. https://doi.org/10.1016/j.neucom.2016.12.092</bibtext> </blist> <blist> <bibtext> Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1 (8), 9.</bibtext> </blist> <blist> <bibtext> Radiuk, P. M. (2017). Impact of training set batch size on the performance of convolutional neural networks for diverse datasets. Information Technology and Management Science, 20 (1), 20 – 24. https://doi.org/10.1515/itms-2017-0003</bibtext> </blist> <blist> <bibtext> Ribeiro, M. T., Singh, S., Guestrin, C. (2016). "why should i trust you?" explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135 – 1144.</bibtext> </blist> <blist> <bibtext> Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., Clopath, C., Costa, R. P., de Berker, A., Ganguli, S., Gillon, C. J., Hafner, D., Kepecs, A., Kriegeskorte, N., Latham, P., Lindsay, G. W., Miller, K. D., Naud, R., Pack, C. C., ... Kording, K. P.,. (2019). A deep learning framework for neuroscience. Nature Neuroscience, 22 (11), 1761 – 1770. https://doi.org/10.1038/s41593-019-0520-2</bibtext> </blist> <blist> <bibtext> Ruby, U., &amp; Yendapalli, V. (2020). Binary cross entropy with deep learning technique for image classification. International Journal of Advanced Trends in Computer Science and Engineering, 9 (4), 5393–5397. https://doi.org/10.30534/ijatcse/2020/175942020</bibtext> </blist> <blist> <bibtext> Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1 (5), 206 – 215. https://doi.org/10.1038/s42256-019-0048-x</bibtext> </blist> <blist> <bibtext> Rumelhart, D. E., Hinton, G. E., Williams, R. J., et al. (1985). Learning internal representations by error propagation.</bibtext> </blist> <blist> <bibtext> Rusiecki, A. (2019). Trimmed categorical cross-entropy for deep learning with label noise. Electronics Letters, 55 (6), 319 – 320. https://doi.org/10.1049/el.2018.7980</bibtext> </blist> <blist> <bibtext> Sagae, K. (2021). Tracking child language development with neural network language models. Frontiers in Psychology, 12, 674402. https://doi.org/10.3389/fpsyg.2021.674402</bibtext> </blist> <blist> <bibtext> Sazli, M. H. (2006). A brief review of feed-forward neural networks. Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering, 50 (01), 11 – 17.</bibtext> </blist> <blist> <bibtext> Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks: The Official Journal of the International Neural Network Society, 61, 85 – 117. https://doi.org/10.1016/j.neunet.2014.09.003</bibtext> </blist> <blist> <bibtext> Snoek, J., Larochelle, H., &amp; Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. In Proceedings of the 26th International Conference on Neural Information Processing Systems, 2951 – 2959.</bibtext> </blist> <blist> <bibtext> Su, C., Xu, Z., Pathak, J., &amp; Wang, F. (2020). Deep learning in mental health outcome research: A scoping review. Translational Psychiatry, 10 (1), 116. https://doi.org/10.1038/s41398-020-0780-3</bibtext> </blist> <blist> <bibtext> Sundararajan, M., Taly, A., Yan, Q. (2017). Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning, 3319 – 3328.</bibtext> </blist> <blist> <bibtext> Suominen, A., &amp; Toivanen, H. (2016). Map of science with topic modeling: Comparison of unsupervised learning and human-assigned subject classification. Journal of the Association for Information Science and Technology, 67 (10), 2464 – 2476. https://doi.org/10.1002/asi.23596</bibtext> </blist> <blist> <bibtext> Tadesse, M. M., Lin, H., Xu, B., &amp; Yang, L. (2019). Detection of suicide ideation in social media forums using deep learning. Algorithms, 13 (1), 7. https://doi.org/10.3390/a13010007</bibtext> </blist> <blist> <bibtext> Urban, C. J., &amp; Gates, K. M. (2021). Deep learning: A primer for psychologists. Psychological Methods, 26 (6), 743 – 773. https://doi.org/10.1037/met0000374</bibtext> </blist> <blist> <bibtext> Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser., &amp; Polosukhin, I. (2017). Attention is all you need. In Proceedings of the 26th International Conference on Neural Information Processing Systems, 2951–2959.</bibtext> </blist> <blist> <bibtext> Venables, W. N., Ripley, B. D. (2002). Modern applied statistics with s (Fourth) [ISBN 0-387-95457-0]. Springer. https://<ulink href="http://www.stats.ox.ac.uk/pub/MASS4/">www.stats.ox.ac.uk/pub/MASS4/</ulink></bibtext> </blist> <blist> <bibtext> Wilson, D. R., &amp; Martinez, T. R. (2003). The general inefficiency of batch training for gradient descent learning. Neural Networks: The Official Journal of the International Neural Network Society, 16 (10), 1429 – 1451. https://doi.org/10.1016/S0893-6080(03)00138-2</bibtext> </blist> <blist> <bibtext> Xu, R., &amp; Wunsch, D. (2005). Survey of clustering algorithms. IEEE Transactions on Neural Networks, 16 (3), 645 – 678. https://doi.org/10.1109/TNN.2005.845141</bibtext> </blist> <blist> <bibtext> Yarkoni, T., &amp; Westfall, J. (2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 12 (6), 1100 – 1122. https://doi.org/10.1177/1745691617693393</bibtext> </blist> <blist> <bibtext> Yu, T., &amp; Zhu, H. (2020). Hyper-parameter optimization: A review of algorithms and applications. arXiv preprint arXiv:2003.05689</bibtext> </blist> <blist> <bibtext> Zhao, R., Wang, D., Yan, R., Mao, K., Shen, F., &amp; Wang, J. (2018). Machine health monitoring using local feature-based gated recurrent unit networks. IEEE Transactions on Industrial Electronics, 65 (2), 1539 – 1548. https://doi.org/10.1109/TIE.2017.2733438</bibtext> </blist> <blist> <bibtext> Zheng, A., &amp; Casari, A. (2018). Feature engineering for machine learning: Principles and techniques for data scientists. O'Reilly Media, Inc.</bibtext> </blist> </ref> <ref id="AN0193318148-36"> <title> Footnotes </title> <blist> <bibtext> The dataset and the code are available at https://osf.io/pwr8t.</bibtext> </blist> </ref> <aug> <p>By Lingbo Tong and Zhiyong Zhang</p> <p>Reported by Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib10" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib73" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib42" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib26" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib40" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib52" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib66" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib35" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib75" firstref="ref10"></nolink> <nolink nlid="nl10" bibid="bib48" firstref="ref11"></nolink> <nolink nlid="nl11" bibid="bib78" firstref="ref12"></nolink> <nolink nlid="nl12" bibid="bib38" firstref="ref15"></nolink> <nolink nlid="nl13" bibid="bib84" firstref="ref16"></nolink> <nolink nlid="nl14" bibid="bib79" firstref="ref17"></nolink> <nolink nlid="nl15" bibid="bib56" firstref="ref18"></nolink> <nolink nlid="nl16" bibid="bib37" firstref="ref19"></nolink> <nolink nlid="nl17" bibid="bib72" firstref="ref22"></nolink> <nolink nlid="nl18" bibid="bib44" firstref="ref25"></nolink> <nolink nlid="nl19" bibid="bib87" firstref="ref26"></nolink> <nolink nlid="nl20" bibid="bib27" firstref="ref27"></nolink> <nolink nlid="nl21" bibid="bib85" firstref="ref33"></nolink> <nolink nlid="nl22" bibid="bib11" firstref="ref39"></nolink> <nolink nlid="nl23" bibid="bib12" firstref="ref41"></nolink> <nolink nlid="nl24" bibid="bib13" firstref="ref42"></nolink> <nolink nlid="nl25" bibid="bib14" firstref="ref43"></nolink> <nolink nlid="nl26" bibid="bib24" firstref="ref44"></nolink> <nolink nlid="nl27" bibid="bib15" firstref="ref45"></nolink> <nolink nlid="nl28" bibid="bib16" firstref="ref46"></nolink> <nolink nlid="nl29" bibid="bib17" firstref="ref48"></nolink> <nolink nlid="nl30" bibid="bib18" firstref="ref49"></nolink> <nolink nlid="nl31" bibid="bib19" firstref="ref50"></nolink> <nolink nlid="nl32" bibid="bib20" firstref="ref51"></nolink> <nolink nlid="nl33" bibid="bib21" firstref="ref52"></nolink> <nolink nlid="nl34" bibid="bib22" firstref="ref53"></nolink> <nolink nlid="nl35" bibid="bib23" firstref="ref54"></nolink> <nolink nlid="nl36" bibid="bib25" firstref="ref56"></nolink> <nolink nlid="nl37" bibid="bib28" firstref="ref59"></nolink> <nolink nlid="nl38" bibid="bib29" firstref="ref60"></nolink> <nolink nlid="nl39" bibid="bib30" firstref="ref61"></nolink> <nolink nlid="nl40" bibid="bib31" firstref="ref62"></nolink> <nolink nlid="nl41" bibid="bib32" firstref="ref63"></nolink> <nolink nlid="nl42" bibid="bib33" firstref="ref65"></nolink> <nolink nlid="nl43" bibid="bib34" firstref="ref66"></nolink> <nolink nlid="nl44" bibid="bib36" firstref="ref68"></nolink> <nolink nlid="nl45" bibid="bib50" firstref="ref70"></nolink> <nolink nlid="nl46" bibid="bib51" firstref="ref73"></nolink> <nolink nlid="nl47" bibid="bib61" firstref="ref74"></nolink> <nolink nlid="nl48" bibid="bib57" firstref="ref76"></nolink> <nolink nlid="nl49" bibid="bib49" firstref="ref77"></nolink> <nolink nlid="nl50" bibid="bib81" firstref="ref82"></nolink> <nolink nlid="nl51" bibid="bib82" firstref="ref84"></nolink> <nolink nlid="nl52" bibid="bib64" firstref="ref89"></nolink> <nolink nlid="nl53" bibid="bib39" firstref="ref91"></nolink> <nolink nlid="nl54" bibid="bib62" firstref="ref92"></nolink> <nolink nlid="nl55" bibid="bib74" firstref="ref98"></nolink> <nolink nlid="nl56" bibid="bib41" firstref="ref99"></nolink> <nolink nlid="nl57" bibid="bib58" firstref="ref101"></nolink> <nolink nlid="nl58" bibid="bib45" firstref="ref104"></nolink> <nolink nlid="nl59" bibid="bib65" firstref="ref107"></nolink> <nolink nlid="nl60" bibid="bib68" firstref="ref108"></nolink> <nolink nlid="nl61" bibid="bib54" firstref="ref111"></nolink> <nolink nlid="nl62" bibid="bib47" firstref="ref113"></nolink> <nolink nlid="nl63" bibid="bib76" firstref="ref115"></nolink> <nolink nlid="nl64" bibid="bib69" firstref="ref116"></nolink> <nolink nlid="nl65" bibid="bib43" firstref="ref117"></nolink> <nolink nlid="nl66" bibid="bib60" firstref="ref121"></nolink> <nolink nlid="nl67" bibid="bib71" firstref="ref122"></nolink> <nolink nlid="nl68" bibid="bib86" firstref="ref124"></nolink> <nolink nlid="nl69" bibid="bib80" firstref="ref128"></nolink> <nolink nlid="nl70" bibid="bib63" firstref="ref131"></nolink> <nolink nlid="nl71" bibid="bib59" firstref="ref133"></nolink> <nolink nlid="nl72" bibid="bib53" firstref="ref134"></nolink> <nolink nlid="nl73" bibid="bib46" firstref="ref135"></nolink> <nolink nlid="nl74" bibid="bib55" firstref="ref136"></nolink> <nolink nlid="nl75" bibid="bib67" firstref="ref139"></nolink> <nolink nlid="nl76" bibid="bib70" firstref="ref141"></nolink> <nolink nlid="nl77" bibid="bib83" firstref="ref143"></nolink> <nolink nlid="nl78" bibid="bib77" firstref="ref147"></nolink> |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: ED679891 AccessLevel: 3 PubType: Report PubTypeId: report PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Neural Network Analysis of Psychological Data: A Step-by-Step Guide – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Lingbo+Tong%22">Lingbo Tong</searchLink><br /><searchLink fieldCode="AR" term="%22Zhiyong+Zhang%22">Zhiyong Zhang</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Grantee+Submission%22"><i>Grantee Submission</i></searchLink>. 2026. – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 23 – Name: DatePubCY Label: Publication Date Group: Date Data: 2026 – Name: SourceSuprt Label: Sponsoring Agency Group: SrcSuprt Data: Institute of Education Sciences (ED) – Name: NumberContract Label: Contract Number Group: NumCntrct Data: R305D210023 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Reports - Evaluative – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Data%22">Data</searchLink><br /><searchLink fieldCode="DE" term="%22Psychological+Studies%22">Psychological Studies</searchLink><br /><searchLink fieldCode="DE" term="%22Network+Analysis%22">Network Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Analysis%22">Data Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Statistical+Inference%22">Statistical Inference</searchLink><br /><searchLink fieldCode="DE" term="%22Regression+%28Statistics%29%22">Regression (Statistics)</searchLink> – Name: DOI Label: DOI Group: ID Data: 10.1080/00273171.2025.2587379 – Name: Abstract Label: Abstract Group: Ab Data: Artificial neural networks (ANN) have attracted increasing attention in the field of psychology. With the availability of software programs, the wide application of ANN becomes possible. However, without a firm understanding of the basics of the ANN, issues can easily arise. This article presents a step-by-step guide for implementing a feed-forward neural network (FNN) on a psychological data set to illustrate the critical steps in building, estimating, and interpreting a neural network model. We start with a concrete example of a basic 3-layer FNN, illustrating the core concepts, the matrix representation, and the whole optimization process. By adjusting parameters and changing the model structure, we examine their effects on model performance. Then, we introduce accessible methods for interpreting model results and making inferences. Through the guide, we hope to help researchers avoid common problems in applying neural network models and machine learning methods in general. [This is the online first version of an article published in "Multivariate Behavioral Research." This work was also supported by the Lucy Family Institute for Data and Society and Notre Dame Global.] – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: CodeSource Label: IES Funded Group: SrcInfo Data: Yes – Name: DateEntry Label: Entry Date Group: Date Data: 2026 – Name: AN Label: Accession Number Group: ID Data: ED679891 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED679891 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1080/00273171.2025.2587379 Languages: – Text: English PhysicalDescription: Pagination: PageCount: 23 Subjects: – SubjectFull: Artificial Intelligence Type: general – SubjectFull: Data Type: general – SubjectFull: Psychological Studies Type: general – SubjectFull: Network Analysis Type: general – SubjectFull: Data Analysis Type: general – SubjectFull: Models Type: general – SubjectFull: Statistical Inference Type: general – SubjectFull: Regression (Statistics) Type: general Titles: – TitleFull: Neural Network Analysis of Psychological Data: A Step-by-Step Guide Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Lingbo Tong – PersonEntity: Name: NameFull: Zhiyong Zhang IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2026 Titles: – TitleFull: Grantee Submission Type: main |
| ResultId | 1 |
