View in EDS HTML Full Text PDF Full Text

Facial Expression Recognition for Probing Students' Emotional Engagement in Science Learning

Saved in:
Bibliographic Details
Title: Facial Expression Recognition for Probing Students' Emotional Engagement in Science Learning
Language: English
Authors: Xiaoyu Tang, Yayun Gong, Yang Xiao, Jianwen Xiong, Lei Bao (ORCID 0000-0003-3348-4198)
Source: Journal of Science Education and Technology. 2025 34(1):13-30.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 18
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Descriptors: Physics, Science Instruction, Nonverbal Communication, Science Achievement, Artificial Intelligence, Emotional Response, Affective Measures, Arousal Patterns, Predictive Validity, Learner Engagement, Teaching Methods, Instructional Effectiveness
DOI: 10.1007/s10956-024-10143-7
ISSN: 1059-0145
1573-1839
Abstract: Student engagement in science classroom is an essential element for delivering effective instruction. However, the popular method for measuring students' emotional learning engagement (ELE) relies on self-reporting, which has been criticized for possible bias and lacking fine-grained time solution needed to track the effects of short-term learning interactions. Recent research suggests that students' facial expressions may serve as an external representation of their emotions in learning. Accordingly, this study proposes a machine learning method to efficiently measure students' ELE in real classroom. Specifically, a facial expression recognition system based on a multiscale perception network (MP-FERS) was developed by combining the pleasure-displeasure, arousal-nonarousal, and dominance-submissiveness (PAD) emotion models. Data were collected from videos of six physics lessons with 108 students. Meanwhile, students' academic records and self-reported learning engagement were also collected. The results show that students' ELE measured by MP-FERS was a significant predictor of academic achievement and a better indicator of true learning status than self-reported ELE. Furthermore, MP-FERS can provide fine-grained time resolution on tracking the changes in students' ELE in response to different teaching environments such as teacher-centered or student-centered classroom activities. The results of this study demonstrate the validity and utility of MP-FERS in studying students' emotional learning engagement.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1460783
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwEoouLfFsEkvNbNbiUHoCXSAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDB7MfMJMxwyRvx6QAgIBEICBmxVLH-Imp1lDJPy7KrnsZlRU_QJEvULHyFgLPY9QDY66-AW3ZBKMWcTdsgUxiGCJezlc7v-mhnGUE4vpQsM3z1x4XsPtwR_mpurq4-D7pyUVD3Q-PmXZ_JnOOiRqeQyRSpAe1QCeBNoPibRL6GLuDoukWZBsR1ikZN36Nqaaw5pvMuHCv1aYPIfGZt0i3-gDzi_fB-oqLoX3B7LY
Text:
  Availability: 1
  Value: <anid>AN0183076991;4n601feb.25;2025Feb19.04:02;v2.2.500</anid> <title id="AN0183076991-1">Facial Expression Recognition for Probing Students' Emotional Engagement in Science Learning </title> <p>Student engagement in science classroom is an essential element for delivering effective instruction. However, the popular method for measuring students' emotional learning engagement (ELE) relies on self-reporting, which has been criticized for possible bias and lacking fine-grained time solution needed to track the effects of short-term learning interactions. Recent research suggests that students' facial expressions may serve as an external representation of their emotions in learning. Accordingly, this study proposes a machine learning method to efficiently measure students' ELE in real classroom. Specifically, a facial expression recognition system based on a multiscale perception network (MP-FERS) was developed by combining the pleasure-displeasure, arousal-nonarousal, and dominance-submissiveness (PAD) emotion models. Data were collected from videos of six physics lessons with 108 students. Meanwhile, students' academic records and self-reported learning engagement were also collected. The results show that students' ELE measured by MP-FERS was a significant predictor of academic achievement and a better indicator of true learning status than self-reported ELE. Furthermore, MP-FERS can provide fine-grained time resolution on tracking the changes in students' ELE in response to different teaching environments such as teacher-centered or student-centered classroom activities. The results of this study demonstrate the validity and utility of MP-FERS in studying students' emotional learning engagement.</p> <p>Keywords: Student engagement; Emotional learning engagement; Facial expression recognition; Science teaching; Education Curriculum and Pedagogy Specialist Studies In Education Psychology and Cognitive Sciences Psychology</p> <p>Supplementary Information The online version contains supplementary material available at https://doi.org/10.1007/s10956-024-10143-7.</p> <hd id="AN0183076991-2">Introduction</hd> <p>As an essential indicator of the impact of reforms in science education, effective teaching has received much attention from the science education community. Currently, explanations of effective teaching can be classified into three intertwined orientations: the importance of teachers' behavioral characteristics in encouraging and facilitating student learning (Blomeke et al., [<reflink idref="bib11" id="ref1">11</reflink>]; Joshi & Bhaskar, [<reflink idref="bib45" id="ref2">45</reflink>]); the importance of meeting students' needs and interests to engage them in the classroom and achieve positive learning outcomes (Elias et al., [<reflink idref="bib26" id="ref3">26</reflink>]; Kennedy, [<reflink idref="bib49" id="ref4">49</reflink>]); and the effectiveness of teacher–student interactions (Sun et al., [<reflink idref="bib95" id="ref5">95</reflink>]; Xintong et al., [<reflink idref="bib108" id="ref6">108</reflink>]). Regardless of the orientation, the goal is to promote effective learning. The first two orientations prioritize external conditions, while the third focuses on internal psychological conditions. Consequently, students' effective learning should be intrinsically motivated to construct knowledge and learning transfer with appropriate guidance and learning activities and promote positive emotional experiences.</p> <p>The different emphases of these three teaching orientations have led to several notable trends in current classroom teaching styles: (<reflink idref="bib1" id="ref7">1</reflink>) teacher-centered teaching, which emphasizes teacher-directed behaviors and verbal expressions to promote students' understanding (Kateřina, [<reflink idref="bib48" id="ref8">48</reflink>]); (<reflink idref="bib2" id="ref9">2</reflink>) student-centered teaching, which emphasizes the active construction of knowledge through students' hands-on participation in learning activities (Eva & Kathleen, [<reflink idref="bib28" id="ref10">28</reflink>]); and (<reflink idref="bib3" id="ref11">3</reflink>) interactive teaching, which emphasizes the interaction between teacher guidance and student activities (Howe et al., [<reflink idref="bib42" id="ref12">42</reflink>]) and promotes the construction of knowledge through both subjects' joint efforts. Teachers' behavioral characteristics in different teaching styles directly impact students' emotional learning engagement and subsequently influence their learning outcomes. For example, the extended use of the lecture mode may lead to student boredom and loss of attention from the classroom. Emotional learning engagement constitutes students' response to the teacher's teaching behavior and serves not only as a prerequisite for learning outcomes but also as an important indicator of the effectiveness of teaching itself.</p> <p>Current educational trends increasingly emphasize the role of emotional learning engagement in teaching and learning. For example, in a study on students' learning effectiveness and core literacy in the UK, measuring students' enjoyment of learning was the focus (Office for Standards in Education, [<reflink idref="bib71" id="ref13">71</reflink>]). Canada's education policy promotes a focus on students' interests and performances in learning (Ontario, [<reflink idref="bib73" id="ref14">73</reflink>]). Moreover, studies from China suggest that teachers should evaluate students' learning through their emotional performance, such as their interest and participation in daily learning activities (China, [<reflink idref="bib18" id="ref15">18</reflink>]). These results show that using students' emotional engagement as a factor for evaluating the effectiveness of classroom teaching has become a significant trend in education assessment. Psychological studies have also shown that positive emotions, such as concentration and happiness, can promote students' learning efficiency. In contrast, negative emotions, such as boredom and anxiety, can decrease intellectual development (Sun et al., [<reflink idref="bib94" id="ref16">94</reflink>]). Moreover, it has been found in numerous studies that classroom emotion and interest are key factors in science learning retention (Prescod et al., [<reflink idref="bib81" id="ref17">81</reflink>]; Sadler et al., [<reflink idref="bib87" id="ref18">87</reflink>]; Schelfhout et al., [<reflink idref="bib88" id="ref19">88</reflink>]). Therefore, importance needs to be attributed to students' emotional changes in the classroom, as such changes can constantly remind teachers to make timely adjustments to their teaching strategies to keep students actively engaged in science learning.</p> <p>In the literature, the commonly used measures of student engagement are self-reports and structured observations, which could be biased in measuring implicit emotional engagement. In addition, structured observations are usually designed with a specific observation plan and purpose, which limits the measurement to be single-angled and subjective, and information on students' emotional experiences is also limited (Mikeska et al., [<reflink idref="bib64" id="ref20">64</reflink>]). Moreover, although self-reports can reveal students' psychological feelings more directly, they are often widely inaccurate in the emotional dimension, especially in lower grades (Ben-Eliyahu et al., [<reflink idref="bib10" id="ref21">10</reflink>]). Furthermore, this uncertainty may be exacerbated by the time lag in measurement and the influence of social expectations. In addition, analyzing data from structured observations and self-reports is labor intensive and can be subjectively biased. Therefore, finding a machine-based objective measuring method could be invaluable for advancing research in this area. The recent emergence of facial expression recognition technology has led to the development of a promising approach (Liu et al., [<reflink idref="bib57" id="ref22">57</reflink>]) that can automatically perform feature extraction and expression recognition on facial images (Mollahosseini et al., [<reflink idref="bib65" id="ref23">65</reflink>]). Motivated by recent work, this study develops a multiscale perception facial expression recognition system (MP-FERS) for measuring students' emotional learning engagement and validates the measurement outcomes of the MP-FERS using a mixed research approach.</p> <hd id="AN0183076991-3">Literature Review</hd> <p></p> <hd id="AN0183076991-4">Student Engagement as a Multidimensional Construct</hd> <p>Many studies have demonstrated that student engagement is a multidimensional construct and can significantly impact students' academic achievement (Engels et al., [<reflink idref="bib27" id="ref24">27</reflink>]; Muenks et al., [<reflink idref="bib66" id="ref25">66</reflink>]). In general, student engagement includes behavioral, emotional, and cognitive engagement (Fredricks et al., [<reflink idref="bib29" id="ref26">29</reflink>]). Behavioral engagement reflects aspects of student learning behaviors, including answering questions, participating in class discussions, and completing expected tasks (Sedova et al., [<reflink idref="bib90" id="ref27">90</reflink>]; Wang et al., [<reflink idref="bib103" id="ref28">103</reflink>]). Cognitive engagement is not directly observable and reflects the degree to which students think about or focus on learning activities (Greene, [<reflink idref="bib36" id="ref29">36</reflink>]). This engagement is mainly involved in the mental effort and cognitive strategies used for understanding knowledge (Connell & Wellborn, [<reflink idref="bib20" id="ref30">20</reflink>]; Newmann, [<reflink idref="bib69" id="ref31">69</reflink>]; Olivier et al., [<reflink idref="bib72" id="ref32">72</reflink>]).</p> <p>Compared to the first two types of engagement, emotional engagement is an implicit psychological state. Since this study focuses on students' emotional engagement in the classroom, it is further defined as emotional learning engagement (ELE), which specifically refers to students' emotional responses to the learning process and classroom environment, including interest and belonging (Connell & Wellborn, [<reflink idref="bib20" id="ref33">20</reflink>]; Skinner & Belmont, [<reflink idref="bib93" id="ref34">93</reflink>]). As part of the dyadic interaction between a learner and a learning activity, ELE is present throughout the learning process (Ben-Eliyahu et al., [<reflink idref="bib10" id="ref35">10</reflink>]). ELE aligns well with research on flow, which refers to learners feeling positive emotions, losing time, and becoming fully immersed during a learning activity (Nakamura & Csikszentmihalyi, [<reflink idref="bib68" id="ref36">68</reflink>]). Flow is often used to describe high-quality ELE. Both situational interests caused by the specific characteristics of classroom activities and personal interests aroused by the willingness to undertake challenging tasks can equip students with a mindset directed toward classroom tasks (Renninger et al., [<reflink idref="bib86" id="ref37">86</reflink>]), which constitutes the psychological basis of classroom experiences. Numerous studies have demonstrated the importance of emotions. Positively activated emotions (e.g., joy, anticipation) may lead to higher behavioral and cognitive engagement (Linnenbrink, [<reflink idref="bib53" id="ref38">53</reflink>]; Pekrun et al., [<reflink idref="bib77" id="ref39">77</reflink>]), while negatively activated emotions (e.g., confusion) may lead to more in-depth inquiry about the learning materials (D'Mello & Graesser, [<reflink idref="bib22" id="ref40">22</reflink>]). The deactivations of emotions (e.g., fatigue) reflect a negative state of being absent from classroom activities, which can lead to a lack of psychological affiliation, causing behavioral and cognitive engagement burnout (Linnenbrink, [<reflink idref="bib53" id="ref41">53</reflink>]; Pekrun et al., [<reflink idref="bib78" id="ref42">78</reflink>]). Notably, deactivated neutral emotions (e.g., boredom) reflect a tendency to detach from ongoing activity and cannot be ignored when modeling learning engagement (Ben-Eliyahu et al., [<reflink idref="bib10" id="ref43">10</reflink>]).</p> <p>In summary, the literature has demonstrated that ELE can play a vital role in student engagement by influencing behavioral and cognitive engagement from a psychological perspective, which focuses on emotional dimensions such as interest, pleasure, and enjoyment. Thus, this study focuses on measuring ELE and its interactions with the learning environment.</p> <hd id="AN0183076991-5">The Influence of ELE on Academic Achievement</hd> <p>Emotions are ubiquitous in academic settings (e.g., emotions such as enjoyment, anger, anxiety, and boredom that arise during the learning process), and they can profoundly impact students' academic engagement and performance (Pekrun & Linnenbrink-Garcia, [<reflink idref="bib80" id="ref44">80</reflink>]). Evidence shows that negative emotions such as anger and sadness are negatively associated with achievement (Hernández et al., [<reflink idref="bib41" id="ref45">41</reflink>]), while positive emotions such as enjoyment are positively associated with achievement (Pekrun & Linnenbrink-Garcia, [<reflink idref="bib80" id="ref46">80</reflink>]). Recent studies have found a feedback loop between emotion and achievement over time (Pekrun et al., [<reflink idref="bib79" id="ref47">79</reflink>]; Putwain et al., [<reflink idref="bib82" id="ref48">82</reflink>], [<reflink idref="bib83" id="ref49">83</reflink>]). For example, higher enjoyment and lower boredom predict greater subsequent achievement, and, in turn, greater academic achievement predicts subsequent greater enjoyment and lower boredom. This suggests that emotion and academic achievement are consistently and tightly intertwined. These empirical studies revealed the feasibility of using student engagement as an indicator of effective teaching (Reinhold et al., [<reflink idref="bib84" id="ref50">84</reflink>]).</p> <p>ELE can have complex and extensive influences on academic achievement because it provides a critical psychological foundation for learning. ELE can influence both behavioral and cognitive engagement, which can further influence academic achievement (Geertshuis, [<reflink idref="bib33" id="ref51">33</reflink>]; Liu et al., [<reflink idref="bib54" id="ref52">54</reflink>], [<reflink idref="bib56" id="ref53">56</reflink>]). For example, students with positive emotions are more likely to devote time and energy to learning and can prevent themselves from possible academic burnout (González-Romá et al., [<reflink idref="bib35" id="ref54">35</reflink>]). Thus, these students may show more sustained behavioral engagement and are more likely to deal effectively with learning difficulties (Wang & Eccles, [<reflink idref="bib102" id="ref55">102</reflink>]). The control-value theory explains that emotions determine the use of cognitive resources and learning strategies, as well as motivation, to influence achievement (Meinhardt & Pekrun, [<reflink idref="bib63" id="ref56">63</reflink>]; Pekrun, [<reflink idref="bib76" id="ref57">76</reflink>]). Positive emotions (e.g., enjoyment) retain cognitive resources, increase interest and motivation, and promote flexible and deep learning strategies, leading to a better likelihood for academic success. Conversely, negative emotions (anger, sadness) may induce irrelevant thinking, reduce cognitive resources, disrupt attentional focus, and prevent the systematic use of deep learning strategies, all of which are detrimental factors to academic progress (Kuhbandner et al., [<reflink idref="bib50" id="ref58">50</reflink>]).</p> <p>Overall, student engagement is dynamically related internally, with emotion being the most foundational factor in academic achievement. ELE influences the formation of motivation, interest, and attention, thus promoting sustainable behavioral engagement (Yang et al., [<reflink idref="bib110" id="ref59">110</reflink>]). Behavioral engagement, in turn, influences cognitive engagement through different learning modes, ultimately affecting learning outcomes (Chi & Wylie, [<reflink idref="bib17" id="ref60">17</reflink>]; Yang et al., [<reflink idref="bib110" id="ref61">110</reflink>]). These studies have provided strong evidence demonstrating the predictive role of emotional engagement in academic achievement, which provides the theoretical and experimental basis for using the MP-FERS to measure student engagement in this study.</p> <hd id="AN0183076991-6">The Influence of Teaching Style on ELE</hd> <p>Numerous studies have demonstrated that teaching style affects students' learning interest and enjoyment (Kang & Keinonen, [<reflink idref="bib47" id="ref62">47</reflink>]). Emotion, as a result of the classroom context, mediates learning outcomes that reflect teaching effectiveness (Schukajlow & Rakoczy, [<reflink idref="bib89" id="ref63">89</reflink>]). As science education shifts from teacher-centered to student-centered, many studies note that student-centered instruction is more likely to increase students' affective interest than traditional teacher-centered instruction (Alimoglu et al., [<reflink idref="bib1" id="ref64">1</reflink>]; Renninger & Bachrach, [<reflink idref="bib85" id="ref65">85</reflink>]; Trobst et al., [<reflink idref="bib97" id="ref66">97</reflink>]). For example, cooperative learning (Sibomana et al., [<reflink idref="bib92" id="ref67">92</reflink>]), game-based approaches (North et al., [<reflink idref="bib70" id="ref68">70</reflink>]), and problem-solving approaches (Taub et al., [<reflink idref="bib96" id="ref69">96</reflink>]) can increase students' enjoyment and positive attitudes toward learning. A cooperative learning environment stimulates student interaction and significantly increases positive emotions (Martínez-Sierra, [<reflink idref="bib59" id="ref70">59</reflink>]). Conversely, game-based instruction conforms to students' instincts, thereby increasing their enjoyment of learning and confidence in success (Battersby et al., [<reflink idref="bib9" id="ref71">9</reflink>]; Byusa et al., [<reflink idref="bib12" id="ref72">12</reflink>]). Further evidence of the facilitative effects of student-centered instruction on student engagement was provided in a mixed study that investigated the perceptions of engagement factors among middle and high school students who varied in their level of science engagement. The researchers found that student-centered instruction significantly influenced ELE, motivational beliefs, and social support (Fredricks et al., [<reflink idref="bib30" id="ref73">30</reflink>]).</p> <p>Summarizing the literature, the positive effect of student-centered instruction on students' ELE is relatively consistent across ages (Areepattamannil, [<reflink idref="bib4" id="ref74">4</reflink>]), grades, and subjects (Capar & Tarim, [<reflink idref="bib14" id="ref75">14</reflink>]). Emotional interest, as a strong predictor of the science learning retention rate, cannot be ignored, and teaching style is an essential factor. This study will further investigate this relationship by using the MP-FERS, which will also provide evidence for the validity of the measurement using this system.</p> <hd id="AN0183076991-7">Measuring ELE by Facial Expression Recognition</hd> <p>Since the abovementioned three types of engagement differ in their degrees of externalization, they are commonly measured by teacher observations and student self-reports (Ben-Eliyahu et al., [<reflink idref="bib10" id="ref76">10</reflink>]; Fredricks et al., [<reflink idref="bib29" id="ref77">29</reflink>]). External behavioral engagement is often measured through teacher observations (Bakker et al., [<reflink idref="bib8" id="ref78">8</reflink>]; Guo et al., [<reflink idref="bib38" id="ref79">38</reflink>]), while cognitive engagement is measured through student self-reports or work samples such as standardized tests and student work (Bakker et al., [<reflink idref="bib8" id="ref80">8</reflink>]). However, measuring ELE is particularly difficult because emotions are implicit. Self-reports are currently the most popular method for measuring emotions. Such measurements rely on behavioral indicators of ELE, which are often difficult for younger students to use to discriminate between the different types of engagement items (Fredricks et al., [<reflink idref="bib29" id="ref81">29</reflink>]). In addition, self-reports are often used as a one-time metric and lack the temporal resolution for tracking ELE variations over time and in connection with specific learning contexts (Park et al., [<reflink idref="bib74" id="ref82">74</reflink>]).</p> <p>To examine classroom effectiveness, we choose to measure ELE, which is influenced by the classroom environment and is characterized by student emotions resulting from classroom elements such as learning content and activities. The traditional measure of ELE usually collects students' subjective feelings and self-evaluations after a course, requiring students to recall the class content and their mental states. This method is both abrupt and subjective (Henrie et al., [<reflink idref="bib40" id="ref83">40</reflink>]), resulting in a measurement of student engagement that may deviate significantly from the real engagement level due to the delay. In addition, students may conceal their low level of engagement due to the influence of social expectations, leading to inaccurate measurements.</p> <p> <emph>The 2017 Horizon Report</emph> (Freeman et al., [<reflink idref="bib31" id="ref84">31</reflink>]) suggested that classroom measurement should focus on using measuring tools to track, analyze, and reflect the learning data from student classroom engagement. With the advancement of technology, information technology can be used to measure student engagement.</p> <p>Mehrabian and Russell ([<reflink idref="bib62" id="ref85">62</reflink>]) research showed that emotional expression consists of 7% words, 38% voice, and 55% facial expressions, which indicates that facial expressions can be an essential avenue for measuring emotion. Although the emotional intensity of facial expressions may vary due to cultural differences (Tsai et al., [<reflink idref="bib98" id="ref86">98</reflink>]), the basic categories of emotions expressed are broadly consistent (Anthony & Nicolas, [<reflink idref="bib3" id="ref87">3</reflink>]). As a result, facial expression recognition (FER) techniques are now widely used. Currently, the information expressed by students' facial expressions is associated with their learning emotions. FER is gradually being applied in teaching environments. Studies have suggested that students' expressions reflect their cognition. Wang et al. ([<reflink idref="bib100" id="ref88">100</reflink>]) studied puzzlement detection using FER. Liaw et al. ([<reflink idref="bib52" id="ref89">52</reflink>]) analyzed changes in students' facial expressions and found a significant relationship with conflict-induced conceptual change, which is valuable for predicting students' learning outcomes. Several studies have also attempted to analyze students' emotions with facial expressions. Chen et al. ([<reflink idref="bib16" id="ref90">16</reflink>]) built detectors of confusion, engagement, and frustration with features extracted from FER. Zhu and Chen ([<reflink idref="bib113" id="ref91">113</reflink>]) constructed a database of students' spontaneous facial expressions and applied it to evaluate emotions during e-learning. Most of these studies have classified emotions based on extracted feature information. Recent research has further quantified emotions as classroom status indicators. For example, Pei and Shan ([<reflink idref="bib75" id="ref92">75</reflink>]) generated students' concentration scores via FER. Shen et al. ([<reflink idref="bib91" id="ref93">91</reflink>]) constructed an engagement equation based on four emotions, namely, neutral, understanding, disgust, and doubt, to generate students' classroom engagement scores. The findings of these studies suggest the feasibility of developing the MP-FERS for classroom evaluation.</p> <p>Although the aforementioned studies achieved acceptable identification accuracy and precision, the quantitative criteria lack theoretical support. Few have explored the strength of the link between engagement measured by computer vision and actual engagement (for example, comparing FER measurements with teacher observations or self-reports). Furthermore, discussions of how FER measurement results work for teaching feedback are lacking (Vanneste et al., [<reflink idref="bib99" id="ref94">99</reflink>]). This study attempts to address these limitations by exploring two research questions, which are discussed next.</p> <p>In this study, the PAD emotional state model proposed by psychologists, such as Mehrabian ([<reflink idref="bib60" id="ref95">60</reflink>]), was used for quantifying ELE through facial recognition. The PAD model describes emotional states through pleasure, arousal, and dominance and uses continuous sampling and recognition of facial expressions to systematically quantify emotion as ELE in real time. It has been demonstrated that almost all the reliable variance in the other 42 emotional response scales can be explained by the PAD emotional state model (Mehrabian, [<reflink idref="bib61" id="ref96">61</reflink>]). Moreover, the PAD model remains valid for facial expression analysis (Cao et al., [<reflink idref="bib13" id="ref97">13</reflink>]; Jia et al., [<reflink idref="bib43" id="ref98">43</reflink>]). Since each emotion has a set of PAD values, the PAD emotion space region can effectively characterize the learner's emotional state. Gilroy et al. ([<reflink idref="bib34" id="ref99">34</reflink>]) established a correlation between flow and emotional state measures through PAD values. As discussed earlier, the flow state represents high-quality ELE, which supports the use of the PAD model as the method for quantifying ELE.</p> <hd id="AN0183076991-8">Research Questions</hd> <p>This research proposes a method for measuring ELE using the MP-FERS and discusses its effectiveness in science classrooms. Specifically, this research aims to answer the following two research questions:</p> <p></p> <ulist> <item> To what extent can MP-FERS produce valid and reliable measures of students' ELE in a real classroom setting?</item> <p></p> <item> How may students' ELE, as measured by the MP-FERS, vary with different teaching activities?</item> </ulist> <hd id="AN0183076991-9">Research Methodology</hd> <p>As discussed previously, measuring ELE is essential but challenging. Using primarily quantitative methods to investigate ELE involves limitations. Therefore, mixed methods were used in this study (Creswell and Clark, [<reflink idref="bib21" id="ref100">21</reflink>]). Mixed methods are suitable for problems where quantitative or qualitative methods are insufficient for developing a comprehensive understanding (Greene, [<reflink idref="bib37" id="ref101">37</reflink>]). This study collected adequate data from multiple sources to better explore the feasibility of using the MP-FERS to measure ELE. With the types of data and analysis methods used, a mixed-method approach was needed for this study.</p> <hd id="AN0183076991-10">Multiscale Perception Facial Expression Recognition System (MP-FERS) for ELE Measurement</hd> <p>Facial expression can provide an essential basis for determining students' ELE. However, teachers have limited capacity to capture changes in each student's facial expressions over time. Accordingly, this research designed a machine learning-based MP-FERS to measure ELE in real time.</p> <hd id="AN0183076991-11">The Organizational Structure of MP-FERS</hd> <p>The organizational structure of the MP-FERS is shown in Fig. 1. First, an HD camera collects real-time videos of students' facial expressions, which are streamed into the sentiment analysis module. Then, the sentiment analysis module predicts the emotions of students' facial images extracted from the video. Finally, the various emotions identified are further analyzed through the engagement measurement module to be quantified based on the pleasure-displeasure, arousal-nonarousal, and dominance-submissiveness (PAD) emotion model proposed by Mehrabian ([<reflink idref="bib60" id="ref102">60</reflink>]). The ELE values are then calculated through an equation to generate a change curve on the display terminal. The continuous sampling and recognition of facial expressions facilitate a more precise capture of students' ELE caused by changes in the classroom environment. Essentially, the MP-FERS primarily measures situational emotions that reflect the effectiveness of instruction.</p> <p>Graph: Fig. 1 The organizational structure of the MP-FERS comprises a camera, sentiment analysis, and engagement measurement modules. MP-FERS is deployed on an edge computing box with an input image size of 640 × 640 and running at a frame rate of 5 fps</p> <hd id="AN0183076991-12">Sentiment Analysis Model of MP-FERS</hd> <p>Facial expressions are among the most potent, natural, and universal signals that humans utilize to convey emotions and intentions. The sentiment analysis model used in this study consists of two steps: (<reflink idref="bib1" id="ref103">1</reflink>) facial image preprocessing and (<reflink idref="bib2" id="ref104">2</reflink>) facial expression recognition for sentiment classification. Preprocessing of facial images is performed mainly by using an HD camera to acquire students' classroom images and then using the OpenCV toolbox to extract students' facial images. FER in the classroom environment may reduce the amount of complete facial feature information due to facial occlusion or pose changes, resulting in a small recognizable range and low accuracy. To address this problem, this study employs a vision transformer (ViT)-based (Dosovitskiy et al., [<reflink idref="bib25" id="ref105">25</reflink>]) multiscale local and global perception network (MLGPN), which learns the local and global representation of expressions and the relationship between different representations at multiple levels, reducing interference from occlusion and pose changes. Its overall network structure is shown in Fig. 2. First, the multiscale local perception unit embeds a channel attention module that guides the network to learn global and local salient features of expression images at different scales. Then, the expression features with multiscale information were analyzed to produce channel and spatial information about the features through the global perception unit composed of the Vit architecture, which adaptively models the global dependencies of learning expression images in different dimensions. Finally, the hierarchical stacking of multiscale local and global perception (MLGP) blocks composed of two types of perception units effectively reduces the influence of pose changes and occlusions.</p> <p>Graph: Fig. 2 The overall architecture of the proposed MP-FERS. The input image is first processed by a backbone network based on a convolutional neural network (CNN) to obtain a 256 × 14 × 14 feature map. The feature map is then passed through the stacked MLGP block layer, where it sequentially passes through multiscale local attention units and global perception units. A classification header module is connected after the output of the last layer of the MLGP block, which consists of fully connected layers for dimensionality reduction, batch normalization, GELU, and other modules. The output sequence is turned into a vector of expression polarity probability distributions, and the expression categories are finally obtained via softmax normalization</p> <p>To better validate the robustness and generalizability of the model, this study was conducted on three large-scale field datasets popularized by FER. The results are shown in Table 1. FERPlus contains 28,709 training images and 3589 test images. The RAF-DB dataset is a representative wild FER dataset with 12,271 training images and 3068 testing images. AffectNet is the largest field dataset for the FER task, with 283,901 training samples and 3500 test samples. Compared to the other methods, the MP-FERS method achieved the best accuracy on all three datasets, as shown in Table 1. The experimental results demonstrate that MP-FERS can learn local nuances and region-global relationships of expressions, effectively reducing the effects of occlusion and pose changes.</p> <p>Table 1 Performance comparison with state-of-the-art methods on three field-scale datasets</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" rowspan="2"><p>Method</p></th><th align="left" rowspan="2"><p>Year</p></th><th align="left" colspan="3"><p>Accuracy (%)</p></th></tr><tr><th align="left"><p><bold>FERPlus</bold></p></th><th align="left"><p><bold>RAF-DB</bold></p></th><th align="left"><p><bold>AffectNet</bold></p></th></tr></thead><tbody><tr><td align="left"><p>SCN (Wang et al., <xref ref-type="bibr" rid="bibr101">2020</xref>)</p></td><td align="left"><p>2020</p></td><td char="." align="char"><p>89.35</p></td><td char="." align="char"><p>88.14</p></td><td char="." align="char"><p>64.53</p></td></tr><tr><td align="left"><p>MA-Net (Zhao et al., <xref ref-type="bibr" rid="bibr112">2021</xref>)</p></td><td align="left"><p>2021</p></td><td char="." align="char"><p>88.99</p></td><td char="." align="char"><p>88.40</p></td><td char="." align="char"><p>64.53</p></td></tr><tr><td align="left"><p>TransFER (Xue et al., <xref ref-type="bibr" rid="bibr109">2021</xref>)</p></td><td align="left"><p>2021</p></td><td char="." align="char"><p>90.83</p></td><td char="." align="char"><p>90.91</p></td><td char="." align="char"><p>66.23</p></td></tr><tr><td align="left"><p>AMP-Net (H. Liu et al.)</p></td><td align="left"><p>2022</p></td><td char="." align="char"><p>-</p></td><td char="." align="char"><p>89.25</p></td><td char="." align="char"><p>64.54</p></td></tr><tr><td align="left"><p>AVT (Jin et al., <xref ref-type="bibr" rid="bibr44">2022</xref>)</p></td><td align="left"><p>2022</p></td><td char="." align="char"><p>90.45</p></td><td char="." align="char"><p>89.93</p></td><td char="." align="char"><p>-</p></td></tr><tr><td align="left"><p>EAC (Zhang et al., <xref ref-type="bibr" rid="bibr111">2022</xref>)</p></td><td align="left"><p>2022</p></td><td char="." align="char"><p>89.64</p></td><td char="." align="char"><p>89.99</p></td><td char="." align="char"><p>65.32</p></td></tr><tr><td align="left"><p>MARN (Chen et al., <xref ref-type="bibr" rid="bibr15">2023</xref>)</p></td><td align="left"><p>2023</p></td><td char="." align="char"><p>89.59</p></td><td char="." align="char"><p>90.03</p></td><td char="." align="char"><p>66.31</p></td></tr><tr><td align="left"><p>MP-FERS</p></td><td align="left"><p><bold>2023</bold></p></td><td char="." align="char"><p><bold>91.34</bold></p></td><td char="." align="char"><p><bold>92.21</bold></p></td><td char="." align="char"><p><bold>66.97</bold></p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0183076991-13">Designing the Measurement Model of ELE</hd> <p>Quantifying emotion scales to produce an engagement index is one of the challenges faced by researchers in the field of emotion measurement. By reviewing studies on ELE, we found that each level of engagement is either associated with or directly explained by emotion. For example, high engagement is expressed as excitement, happiness, and surprise, while disengagement is expressed as tiredness, boredom, and sadness (Altuwairqi et al., [<reflink idref="bib2" id="ref106">2</reflink>]; D'Mello & Graesser, [<reflink idref="bib22" id="ref107">22</reflink>]). The definition of engagement categories is consistent with the description of multidimensional emotions. Hence, this study associates emotion categories with ELE.</p> <p>Next, we quantify the predicted emotional labels of facial expressions through the PAD emotion model in this study. The PAD conceptualizes emotions of pleasure, arousal, and dominance (Mehrabian, [<reflink idref="bib60" id="ref108">60</reflink>]), which are linked to the positive and negative characteristics of emotions, the level of neurophysiological activation, and the individual's state of control over the situation or others, respectively. Unlike discrete emotion models and other dimensional emotion models (Arent, [<reflink idref="bib5" id="ref109">5</reflink>]; Wundt, [<reflink idref="bib107" id="ref110">107</reflink>]), the PAD emotion model describes subjective experiences and maps their relations to external performance and physiological arousal; thus, it is considered more appropriate to describe students' ELE under the influence of classroom situations (Jia et al., [<reflink idref="bib43" id="ref111">43</reflink>]). In expression recognition, human expressions are classified into seven main types: happy, angry, disgusted, fearful, sad, surprised, and neutral. Table 2 shows the mapping values of these emotions in the three dimensions proposed by the Institute of Psychology, Chinese Academy of Sciences.</p> <p>Table 2 The mapping values of seven emotions in each dimension (Liu et al., [<reflink idref="bib58" id="ref112">58</reflink>])</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p><bold>Emotion</bold></p></th><th align="left"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mi mathvariant="bold-italic">P</mi></mrow></math><inline-graphic mime-subtype="GIF" href="10956_2024_10143_Article_IEq1.gif" /></p></th><th align="left"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mi mathvariant="bold-italic">A</mi></mrow></math><inline-graphic mime-subtype="GIF" href="10956_2024_10143_Article_IEq2.gif" /></p></th><th align="left"><p><math xmlns="http://www.w3.org/1998/Math/MathML"><mrow xmlns=""><mi mathvariant="bold-italic">D</mi></mrow></math><inline-graphic mime-subtype="GIF" href="10956_2024_10143_Article_IEq3.gif" /></p></th></tr></thead><tbody><tr><td align="left"><p>Neutral</p></td><td align="left"><p>0</p></td><td align="left"><p>0</p></td><td align="left"><p>0</p></td></tr><tr><td align="left"><p>Happy</p></td><td align="left"><p>2.77</p></td><td align="left"><p>1.21</p></td><td align="left"><p>1.42</p></td></tr><tr><td align="left"><p>Angry</p></td><td align="left"><p> − 1.98</p></td><td align="left"><p>1.10</p></td><td align="left"><p>0.60</p></td></tr><tr><td align="left"><p>Fearful</p></td><td align="left"><p> − 0.93</p></td><td align="left"><p>1.30</p></td><td align="left"><p> − 0.64</p></td></tr><tr><td align="left"><p>Disgusted</p></td><td align="left"><p> − 1.80</p></td><td align="left"><p>0.40</p></td><td align="left"><p>0.67</p></td></tr><tr><td align="left"><p>Sad</p></td><td align="left"><p> − 0.89</p></td><td align="left"><p>0.17</p></td><td align="left"><p> − 0.70</p></td></tr><tr><td align="left"><p>Surprised</p></td><td align="left"><p>1.72</p></td><td align="left"><p>1.71</p></td><td align="left"><p>0.22</p></td></tr></tbody></table> </ephtml> </p> <p>Although the PAD values are useful, they are not integrated into a single engagement measure. To address this, we created a single measure in terms of the PAD values to represent the overall ELE of an individual or the whole class, which is described in Eq. (<reflink idref="bib1" id="ref113">1</reflink>). Since the contribution of the three dimensions to ELE varies across subjects and teaching forms, the weight values of each dimension <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mfenced close=")" open="("><mi>α</mi><mo>,</mo><mi>β</mi><mo>,</mo><mi>γ</mi></mfenced></math> </ephtml> are constantly changing under the attention mechanism. The subscript <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mi>j</mi></math> </ephtml> refers to the number of emotional categories, <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>P</mi><mo>,</mo><mi>A</mi><mo>,</mo><mi>D</mi></mrow></math> </ephtml> refers to the mapping values of the seven emotions in the PAD model (Table 2), and <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>P</mi><mi>j</mi></msub></math> </ephtml> is the probability of each emotion category. <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mn>7</mn></msubsup><mi>P</mi><msub><mi>P</mi><mi>j</mi></msub><mo>,</mo><msubsup><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mn>7</mn></msubsup><mi>A</mi><msub><mi>P</mi><mi>j</mi></msub></mrow></math> </ephtml> , and <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msubsup><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mn>7</mn></msubsup><mi>D</mi><msub><mi>P</mi><mi>j</mi></msub></mrow></math> </ephtml> represent pleasure, arousal, and dominance values, respectively. During the pretraining of the evaluation model, this research established the equation by adjusting the parameters several times so that the machine scores would be the closest to the manual scores.</p> <p>1 <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>E</mi><mo>=</mo><mi>α</mi><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mn>7</mn></munderover><mi>P</mi><msub><mi>P</mi><mi>j</mi></msub><mo>+</mo><mi>β</mi><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mn>7</mn></munderover><mi>A</mi><msub><mi>P</mi><mi>j</mi></msub><mo>+</mo><mi>γ</mi><munderover><mo>∑</mo><mrow><mi>j</mi><mo>=</mo><mn>1</mn></mrow><mn>7</mn></munderover><mi>D</mi><msub><mi>P</mi><mi>j</mi></msub></mrow></math> </ephtml></p> <p>Graph</p> <p>Equation (<reflink idref="bib1" id="ref114">1</reflink>) calculates the ELE at multiple points in time. To evaluate individual students and the whole class, we define <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>E</mi><mi>k</mi></msub></math> </ephtml> as the average ELE of the <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><msup><mrow><mi>k</mi></mrow><mrow><mi mathvariant="italic">th</mi></mrow></msup></math> </ephtml> student in Eq. (<reflink idref="bib2" id="ref115">2</reflink>), which calculates the effect of classroom activities on stimulating individual interest in learning. By combining the <ephtml> <math xmlns="http://www.w3.org/1998/Math/MathML"><msub><mi>E</mi><mi>k</mi></msub></math> </ephtml> of all students, we can calculate the ELE of the whole class as shown in Eq. (<reflink idref="bib3" id="ref116">3</reflink>), which combines the engagement of all students. In the equation, <emph>m</emph> denotes the total number of time points measured during the class, and <emph>n</emph> represents the number of students.</p> <p>2 <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>E</mi><mi>k</mi></msub><mo>=</mo><mfrac><mrow><msubsup><mo>∑</mo><mrow><mi>i</mi><mo>=</mo><mn>1</mn></mrow><mi>m</mi></msubsup><msub><mi>E</mi><mi>i</mi></msub></mrow><mi>m</mi></mfrac></mrow></math> </ephtml></p> <p>Graph</p> <p>3 <ephtml> <math display="block" xmlns="http://www.w3.org/1998/Math/MathML"><mrow><msub><mi>E</mi><mi>t</mi></msub><mo>=</mo><mfrac><mrow><msubsup><mo>∑</mo><mrow><mi>l</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></msubsup><msub><mi>E</mi><mrow><mi mathvariant="italic">kl</mi></mrow></msub></mrow><mi>n</mi></mfrac></mrow></math> </ephtml></p> <p>Graph</p> <hd id="AN0183076991-14">Procedure and Analysis</hd> <p>As discussed in the review, research on facial emotion recognition tools lacks validity confirmation. The validity of the MP-FERS is equivalent to the concept of validity in social science research. Therefore, this study evaluated the validity of the MP-FERS by comparing the MP-FERS outcomes with other measures, including self-reports of engagement and students' academic performance. The overall framework of the measures and comparisons are shown in Fig. 3.</p> <p>Graph: Fig. 3 A mixed-methods approach is used to validate the MP-FERS dataset. This is a triangulated validation model that includes evidence from self-reports, academic achievement, and teaching style, each corroborating MP-FERS results from a different dimension</p> <p>As discussed earlier, objective evidence of ELE should come from students' behaviors in real classroom situations, i.e., the measurement using MP-FERS, which is the core of this study. Due to the greater specificity of the MP-FERS data compared to those collected by traditional test instruments such as questionnaires, we employed criterion-related validity (Cohen et al., [<reflink idref="bib19" id="ref117">19</reflink>]). Criterion validity reflects the degree to which a measurement instrument is valid for measuring or predicting an individual's performance in a given context. It is critical to find suitable criteria that reflect students' ELE. In this study, two criteria were selected for validation.</p> <p>First, self-reports are still regarded as the standard method because they produce perceptive and subjective evidence directly from subjects (Fredricks et al., [<reflink idref="bib29" id="ref118">29</reflink>]). In this study, students' self-reported engagement was measured within the same time frame as that of the MP-FERS, which can serve as a criterion to validate concurrent validity.</p> <p>In addition, evidence of related factors cannot be ignored. As discussed in the literature review, students' ELE is strongly influenced by teaching style and is an effective predictor of academic achievement. For this reason, the degree of student-centeredness in the classroom and student academic achievement are also selected as related criteria to validate the predictive validity of the scale.</p> <p>By analyzing and comparing the three interrelated groups of measures from multiple data sources, mixed methods facilitate the triangulation of the data analysis to establish the validity and reliability of the MP-FERS measurement outcomes (Delahunty et al., [<reflink idref="bib23" id="ref119">23</reflink>]). Since perceptive and subjective evidence strongly predict related evidence, if the ELE measured by the MP-FERS matches subjective evidence and predicts related evidence equally well, a high level of validity of measurements should be demonstrated using the MP-FERS.</p> <hd id="AN0183076991-15">Participants and Context</hd> <p>The study was conducted with 118 eighth- and eleventh-grade students from two middle schools in Guangdong Province, China. These students majored in a science-based curriculum track that included courses in physics, chemistry, and biology. Ten participants did not complete the final test (8.5%); this is a relatively low percentage of missing participants that may not have affected the subsequent analysis (Hair et al., [<reflink idref="bib39" id="ref120">39</reflink>]). Ultimately, 108 students completed the course and reported their engagement and academic achievement. The sample included 62 boys (57%) and 46 girls (43%).</p> <hd id="AN0183076991-16">Study Design and Procedure</hd> <p>This study was conducted in physics classes. We recorded the participants' classroom facial expressions for six 40-min-long lessons. To maximize the ability to measure authentic emotional engagement, the researchers administered self-report questionnaires immediately after each lesson was completed. To measure students' academic achievement, their final exam scores were obtained from the schools. Finally, three researchers majoring in science education analyzed the teaching clips, which were categorized into three teaching styles.</p> <p>Before the study, students and teachers voluntarily agreed to participate and to provide us with all of the requested personal data. The participants were informed that sensitive facial information would not be retained and that all the data would be kept confidential. Official informed consent was obtained following the requirements and policies of the schools and the local ethics committees.</p> <hd id="AN0183076991-17">Additional Measurements and Analysis</hd> <p>In addition to using the MP-FERS for measuring students' ELE, numerous additional measurement methods, including a questionnaire survey for self-reported ELE, course grades for academic achievement, and class video analysis for teaching style, were used in this study.</p> <hd id="AN0183076991-18">Self-Reported ELE</hd> <p>To measure self-reported ELE, the Science Learning Engagement Scale (SLES) developed by Ben-Eliyahu et al. ([<reflink idref="bib10" id="ref121">10</reflink>]) was used. The SLES is a reliable and valid instrument for measuring student engagement and includes emotional, behavioral, and cognitive engagement. The original scale is used to measure student engagement in both formal and informal learning. In this study, only the context of science classroom learning, which included 17 items, was retained. The complete scale is shown in the supplementary material. The three reverse-structured questions (Q3, Q4, Q5) included in the scale were reverse-coded before analyzing the data. A higher score indicates a higher level of engagement.</p> <hd id="AN0183076991-19">Academic Achievement</hd> <p>In this study, academic achievement was evaluated using students' final physics test scores. The questions were assigned by the local municipal education authorities and developed by a panel of senior teachers after two rounds of reviews. The tests included multiple-choice, fill-in-the-blank (short answer), and computational show-work questions, with a total possible score of 100. Since we only measured ELE in six lessons during the semester, the test scores on selected questions that corresponded to the content areas of the six lessons were used for student achievement. Each student's score was normalized.</p> <hd id="AN0183076991-20">Teaching Style</hd> <p>The teaching styles were categorized using the S-T interaction analysis scale, which is a typical method for quantitatively analyzing classroom instruction that quantifies the distribution of teacher behavior (T) and student behavior (S) based on a classroom observation framework (Kaiyue et al., [<reflink idref="bib46" id="ref122">46</reflink>]). This method distinguishes student-centered instruction from teacher-centered instruction by analyzing teacher occupancy (Rt) at each classroom stage (Fu & Zhang, [<reflink idref="bib32" id="ref123">32</reflink>]). According to Li et al. ([<reflink idref="bib51" id="ref124">51</reflink>]), Rt > 0.7 is the lecture type, which is defined as teacher-centered in this study; Rt < 0.3 is the practice type, which is defined as student-centered in this study; and 0.3 < Rt < 0.7 is the interactive type. In this study, the coding method was based on educational information processing technology (Fu & Zhang, [<reflink idref="bib32" id="ref125">32</reflink>]). A period of class time was coded as teacher behavior (T) if it was dominated by the teacher with explanations, demonstrations, media displays, questions, or evaluations. In contrast, a class period was coded as student behavior (S) if it was dominated by students with speech, reading, thinking, discussing, experimenting, or notetaking. According to previous S-T analysis studies (Dong & Ke, [<reflink idref="bib24" id="ref126">24</reflink>]; Liu et al., [<reflink idref="bib55" id="ref127">55</reflink>]), the minimum teaching session length was typically 2 min, while the duration of the behavior was typically less than 3 min. Therefore, the classroom videos were divided into 2- to 3-min segments of teaching clips, which were coded based on the teacher and student behaviors discussed above to determine the teaching styles.</p> <hd id="AN0183076991-21">Results</hd> <p></p> <hd id="AN0183076991-22">Correlation Analysis for Criterion Validity</hd> <p>To establish the criterion validity of the MP-FERS, ELE scores measured from the MP-FERS were compared with self-reported data. In addition, correlations between ELE scores and students' physics test scores were also analyzed. However, since the MP-FERS and self-reports are completely different measures, the absolute scales of the results from the two methods are not directly comparable, but correlations can be used to compare their variances. However, since ELE is believed to be a strong predictor of academic performance, analyzing the correlations among ELE measures and student academic performance can provide useful evidence for establishing the validity of the MP-FERS. Descriptive details of the dataset are included in the supplementary materials. The correlations are provided in Table 3 and discussed next.</p> <p>Table 3 Descriptive statistics and correlations among variables</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" rowspan="2"><p>Variable</p></th><th align="left" rowspan="2"><p>Mean</p></th><th align="left" rowspan="2"><p>SD</p></th><th align="left" colspan="2"><p>Correlation</p></th></tr><tr><th align="left"><p>1</p></th><th align="left"><p>2</p></th></tr></thead><tbody><tr><td align="left"><p>1 ELE from MP-FERS</p></td><td char="." align="char"><p>61.68</p></td><td char="." align="char"><p>8.38</p></td><td char="." align="char"><p>-</p></td><td char="." align="char" /></tr><tr><td align="left"><p>2 ELE from Self-report</p></td><td char="." align="char"><p>87.06</p></td><td char="." align="char"><p>10.18</p></td><td char="." align="char"><p>0.496**</p></td><td char="." align="char"><p>-</p></td></tr><tr><td align="left"><p>3 Physics final test score</p></td><td char="." align="char"><p>54.55</p></td><td char="." align="char"><p>26.15</p></td><td char="." align="char"><p>0.845**</p></td><td char="." align="char"><p>0.479**</p></td></tr></tbody></table> </ephtml> </p> <p>*<emph>p</emph> < 0.05, **<emph>p</emph> < 0.01</p> <p>Self-reports are commonly used as subjective measures of ELE; therefore, we first compared the correlation between ELE scores from the MP-FERS and self-reports. The correlation matrix showed that the ELE scores from the MP-FERS and self-reports were positively and significantly correlated (<emph>r</emph> = 0.496, <emph>p</emph> < 0.01; <emph>R</emph><sups>2</sups> = 0.25). This result suggested that the MP-FERS has appropriate concurrent validity with students' self-reported ELE scores. Accordingly, the ELE measured by the MP-FERS is a suitable indicator of students' emotional state in the classroom.</p> <p>However, the correlation was within the medium range, indicating a moderate level of inconsistency between the subjective and objective measures. In self-reports, many students reported close to perfect scores on the emotional engagement dimension, leading to a small variance in the measurement. However, there were clear differences in students' learning behaviors and facial expressions, as seen in the videos, which indicate that students' self-reports of ELE are likely biased by intended preferences. For example, a significant fraction of students displayed expressions of boredom during the instruction, as seen from the class video, but reported a high level of engagement. Therefore, we need to further analyze the correlations between ELE measures and academic achievement to determine which measure has more predictive value for learning. If the MP-FERS results are more significantly correlated with science achievement than self-reports are, ELE from the MP-FERS is more strongly correlated with student performance. Thus, we can consider the MP-FERS score to be a better predictor of student performance. Two examples of student data are shown in the supplementary materials.</p> <p>As shown in Table 3, the ELE obtained from the MP-FERS was strongly correlated with physics scores (<emph>r</emph> = 0.845, <emph>p</emph> < 0.01; <emph>R</emph><sups>2</sups> = 0.71) and was much greater than the correlation between self-reported ELE and physics scores (<emph>r</emph> = 0.479, <emph>p</emph> < 0.01; <emph>R</emph><sups>2</sups> = 0.23). This result suggested that the MP-FERS score was a better predictor of student performance than self-reports were. This result further reveals the weakness of self-reports, which are subjective and involve a high degree of uncertainty. For example, students may know what teachers expect them to answer and do not want their low level of engagement to be revealed; thus, self-reported ELE can be strongly biased by social expectations. Moreover, self-reports were obtained after the completion of an entire lesson, which means that students may ignore some neutral or mildly negative feelings and still feel good about themselves. The identified limitations of self-reports are consistent with similar concerns in the literature, as discussed in the literature review. Accordingly, we believe that the strong correlation between MP-FERS scores and students' physics test scores demonstrates a higher level of criterion validity for the MP-FERS than for self-reports.</p> <hd id="AN0183076991-23">Applications of MP-FERS to Informing Teaching</hd> <p>The unique advantage of the MP-FERS is that it can provide near real-time measures of ELE, which can be used to inform teaching practices. To explore how the MP-FERS can be applied in real classrooms, we analyzed the relationships between teaching style and students' ELE measured by the MP-FERS. As indicated by related research, teaching style can significantly influence ELE (Fredricks et al., [<reflink idref="bib30" id="ref128">30</reflink>]); therefore, understanding how such influence manifests in a classroom can provide valuable information for teachers to adjust their teaching strategies so that appropriate ELE states can be maintained during the teaching process.</p> <hd id="AN0183076991-24">Student ELE in Different Teaching Styles</hd> <p>To examine the utility of the MP-FERS, we compared the differences between the measured ELE states of students in different teaching styles, including student-centered, interactive, and teacher-centered styles. If the student-centered ELE is significantly higher than the teacher-centered ELE, then we can consider the MP-FERS a practical tool for probing students' emotional responses to classroom teaching and helping teachers improve teaching effectiveness.</p> <p>In the analysis, we divided the collected classroom videos into a total of 139 clips of 2–3 min each and calculated the average ELE of the whole class in each clip. Researchers also reviewed the teaching style of each clip based on teacher and student activities; 49 teacher-centered clips, 52 interactive clips, and 38 student-centered clips were identified. The ELE was measured with MP-FERS using video images from 65 students whose ELE could be identified in a teaching clip. Each student typically had 800–1400 measured ELE data points in a lesson. All the students' data were aggregated to produce the average ELE in video clips of the three different teaching styles. The results are shown in Fig. 4 as violin plots, which combine the features of the boxplot and density plot. The plots were generated using the ggpubr package in the statistical software R. Explanations of the types of information included in the plots are also provided in the supplementary material.</p> <p>Graph: Fig. 4 Emotional learning engagement (ELE) in teacher-centered, interactive, and student-centered styles. The violin plots show the distributions of students' ELE in the three teaching styles, as well as the medians. The p-value from ANOVA was used to compare the differences among all three groups, while the p-values between any two group means were obtained with t-tests to compare the difference between two specific groups</p> <p>As shown in Fig. 4, there was a significant difference in ELE scores across the three teaching styles (<emph>F</emph>(<reflink idref="bib2" id="ref129">2</reflink>,<reflink idref="bib136" id="ref130">136</reflink>) = 3.99, <emph>p</emph> = 0.021; <emph>η</emph><sups>2</sups> = 0.055). The mean ELE score was highest for the student-centered style (<emph>M</emph> = 58.15, <emph>SD</emph> = 7.43) and lowest for the teacher-centered style (<emph>M</emph> = 53.41, <emph>SD</emph> = 6.95), while the ELE score for the interactive style (<emph>M</emph> = 55.59, <emph>SD</emph> = 8.67) was between these two values. Independent <emph>t</emph>-tests indicated that the ELE was significantly greater in the student-centered clip than in the teacher-centered clip (<emph>t</emph> = 3.06, <emph>p</emph> = 0.003; Cohen's <emph>d</emph> = 0.66). However, the ELE in the interactive clip was not significantly different from that in either the student-centered clip (<emph>t</emph> = − 1.47, <emph>p</emph> = 0.15; Cohen's <emph>d</emph> = 0.32) or the teacher-centered clip (<emph>t</emph> = 1.39, <emph>p</emph> = 0.17; Cohen's <emph>d</emph> = 0.28).</p> <p>The results showed that students had higher ELE in teaching clips with a higher degree of student activity, which is consistent with the findings of previous studies (Fredricks et al., [<reflink idref="bib30" id="ref131">30</reflink>]; Renninger & Bachrach, [<reflink idref="bib85" id="ref132">85</reflink>]). Therefore, the MP-FERS measurement of ELE can be considered a convenient and viable tool for probing real-time ELE in teaching and learning.</p> <hd id="AN0183076991-25">ELE for Students at Different Academic Levels</hd> <p>To explore how different teaching styles may impact the emotional reactions of students at different academic levels, the students were sorted into two performance groups: a high-score group (top 50%) and a low-score group (bottom 50%). The two groups' ELE scores for different teaching styles were analyzed and are shown in Table 4.</p> <p>Table 4 Descriptive statistics of ELE measures of the high- and low-performance groups</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" rowspan="2"><p>Teaching style</p></th><th align="left" colspan="2"><p>Mean (standard deviation)</p></th><th align="left" rowspan="2"><p><italic>t value</italic></p></th><th align="left" rowspan="2"><p><italic>p-value</italic></p></th><th align="left" rowspan="2"><p>Cohen's <italic>d</italic></p></th></tr><tr><th align="left"><p><bold>High-score group</bold></p></th><th align="left"><p><bold>Low-score group</bold></p></th></tr></thead><tbody><tr><td align="left"><p>Teacher-centered style</p></td><td align="left"><p>53.36 (12.29)</p></td><td align="left"><p>50.03 (14.77)</p></td><td char="." align="char"><p>2.79</p></td><td char="." align="char"><p>0.005</p></td><td char="." align="char"><p>0.25</p></td></tr><tr><td align="left"><p>Interactive style</p></td><td align="left"><p>55.62 (12.17)</p></td><td align="left"><p>53.84 (15.63)</p></td><td char="." align="char"><p>1.31</p></td><td char="." align="char"><p>0.190</p></td><td char="." align="char"><p>0.13</p></td></tr><tr><td align="left"><p>Student-centered style</p></td><td align="left"><p>58.58 (13.03)</p></td><td align="left"><p>57.96 (13.99)</p></td><td char="." align="char"><p>0.42</p></td><td char="." align="char"><p>0.672</p></td><td char="." align="char"><p>0.05</p></td></tr></tbody></table> </ephtml> </p> <p>One-way ANOVA revealed that ELE score significantly differed among the three teaching styles for both the high-score group (<emph>F</emph>(<reflink idref="bib2" id="ref133">2</reflink>,<reflink idref="bib696" id="ref134">696</reflink>) = 9.57, <emph>p</emph> < 0.001, <emph>η</emph><sups>2</sups> = 0.027) and the low-score group (<emph>F</emph>(<reflink idref="bib2" id="ref135">2</reflink>,<reflink idref="bib609" id="ref136">609</reflink>) = 13.96, <emph>p</emph> < 0.001, <emph>η</emph><sups>2</sups> = 0.04). The ELE in both groups was highest in the student-centered style group and lowest in the teacher-centered style group (see Table 4). This result indicates that the effect of teaching style on students' emotional experience is relatively consistent across performance levels. The overall ELE was significantly higher in the high-score group than in the low-score group (<emph>p</emph> = 0.001). However, among the three teaching styles, the difference between the performance groups was significant only for the teacher-centered style (<emph>t</emph> = 2.79, <emph>p</emph> = 0.005; Cohen's <emph>d</emph> = 0.25), and there were no significant differences between the two groups for either the interactive (<emph>t</emph> = 1.31, <emph>p</emph> = 0.190; Cohen's <emph>d</emph> = 0.13) or student-centered styles (<emph>t</emph> = 0.42, <emph>p</emph> = 0.672; Cohen's <emph>d</emph> = 0.05). This finding suggested that students' emotional responses to interactive and student-centered styles are similar across achievement levels. Both interactive and student-centered teaching styles involve open-exploratory methods, which provide students with more opportunities to express themselves and be engaged in learning. An active classroom atmosphere allows most students to experience a sense of participation and thus present collective engagement. On the other hand, in the passive teacher-centered style, higher performing students often have a better chance of keeping up with the instructor's lectures than weaker students, leading to a more pronounced difference between their engagement levels. In addition, since teacher-centered lectures lack interaction, students' ELE depends heavily on their own intrinsic learning motivations and interest, which are significantly correlated with achievement and therefore can also lead to differences in ELE between performance groups. To summarize, the results suggest that designing appropriate interactive and student-centered activities can be an effective strategy for improving the ELE of the majority of students, regardless of their academic performance levels; therefore, such activities can provide a more inclusive environment for teaching and learning.</p> <hd id="AN0183076991-26">Changes in Student's ELE During a Lesson</hd> <p>One advantage of MP-FERS over traditional self-reports is that the measurements are automatically performed with class videos that can produce near real-time outcomes; this approach can be used to analyze fine-grained teaching interactions for evaluating their effectiveness and extend the ability for real-time feedback to improve the classroom environment toward better learning engagement. To examine whether MP-FERS can provide real-time fine-grained ELE feedback for monitoring and improving teaching interactions, the temporal variation in MP-FERS-measured ELE in one lesson was analyzed. Since students' ELE varied greatly from lesson to lesson due to changes in content and teaching emphasis, we selected one lesson as an example to demonstrate the features and capacities of MP-FERS for real-time measurement. The results are shown in Fig. 5 with a time resolution of 3–5 min. The outcomes were calculated with a moving average of a 5-min window throughout the 40-min class time. Each window in the plot was calculated based on 9 students' data, with approximately 60–170 data points per student in each 5-min time frame. The total sample size for each calculated mean is in the range of 540–1530, which makes the standard errors very small (error bars not shown). For the time frame of each data point plotted, the teaching style was also determined based on the majority of the types of teaching activities.</p> <p>Graph: Fig. 5 The curves of students' emotional learning engagement in a lesson</p> <p>The results showed that both the teaching style and the students' ELE varied substantially throughout the lesson period. The trends in ELE indicate that high-performing students can often maintain a higher and more stable level of ELE, while low-performing students have a lower and more fluctuating ELE during class time. The results also showed that most of the peak engagement stages involved student-centered styles, such as doing exercises (minutes 27–33) and group discussions (minutes 35–37). These findings are consistent with the summative results discussed in the previous section but provide a timeline extension to allow fine-grained resolution for examining the changes in ELE with time and teaching style, which can provide valuable utility for research and teaching. For example, the results show that appropriate interactions can significantly increase students' ELE. During the instruction episode in minutes 27–33, the middle school teacher in the class asked all the students to assume the role of bomb disposal expert and dismantle an explosive device, which had a control circuit fabricated in series and parallel forms. All the students in the class participated in the activity, which stimulated an ELE peak, as shown in Fig. 5. This example illustrates that students' interest in science can increase when they are given roles in which they can take the lead to solve a context-rich problem.</p> <p>The results in Fig. 5 can also provide useful diagnostic information for further analysis of teaching activities. For example, during minutes 15–20, which was a teacher-centered stage, the ELE changes of the high- and low-score groups were vastly different from the states in the remaining time frames. Further analysis of this teaching stage suggested that the teacher was explaining a difficult practice problem in which high-performing students were able to keep up with and respond to teacher. On the other hand, low-performing students were not able to meaningfully follow the discussion and were left out of the interaction loop, which might further lead to frustration among these students. Given this feedback, teachers can improve the design of their teaching by changing to use a more inclusive strategy, such as adding additional "scaffolding" steps, to help develop desirable learning pathways among all students.</p> <hd id="AN0183076991-27">Conclusions and Implications</hd> <p>In this research, we developed a multiscale perception facial expression recognition system (MP-FERS) for measuring students' emotional learning engagement, which was applied to study real classrooms' teaching activities.</p> <p>For the first research question, it was found in this study that ELE measured by the MP-FERS is moderately correlated with ELE measured via self-reports, indicating moderately good concurrent validity. Moreover, the ELE measured by the MP-FERS is more strongly correlated with academic achievement than self-reported ELE, revealing greater predictive validity; these findings are consistent with those of Muñoz-García and Villena-Martínez ([<reflink idref="bib67" id="ref137">67</reflink>]). The results also suggest that the MP-FERS can help address the weakness of self-reports, which are a subjective measure that is likely biased by students' intentions. In contrast, the MP-FERS is an objective measure that cannot be easily concealed by students' intentions. We scrutinized the self-reported and MP-FERS measures, and found that self-reported ELE was generally higher than the ELE measured by the MP-FERS for both high- and low-performance groups. This finding could indicate potential biases in self-reporting, where students may tend to report in a way that aligns with teachers' expectations. The MP-FERS, on the other hand, provides a relatively objective measure that is less likely and harder to be intentionally biased. In addition, the MP-FERS is noninvasive and far more efficient than a questionnaire and provides real-time results at much finer temporal resolutions, which are valuable for research and teaching. In previous research, Whitehill et al. ([<reflink idref="bib105" id="ref138">105</reflink>]) constructed an FER model that discriminates between four levels of engagement, whereas human observers can distinguish between only high and low engagement. Ashwin and Guddeti ([<reflink idref="bib6" id="ref139">6</reflink>]) also found that the machine classification of emotional states provides the same classification as human annotations. All of these outcomes demonstrate the efficiency and potential of FER systems for measuring ELE through students' facial expressions.</p> <p>For the second research question, this study explored how teaching styles impacted students' ELE overall and at high- and low-performance levels. Students are generally more engaged in student-centered and interactive activities. This finding is consistent with previous studies that used self-reports to measure ELE, all of which have demonstrated the positive influence of student-centered teaching on student engagement (Baeten et al., [<reflink idref="bib7" id="ref140">7</reflink>]; Watson et al., [<reflink idref="bib104" id="ref141">104</reflink>]). From Fig. 4, the distribution of ELE across teaching styles reveals that ELE is concentrated at a lower level for teacher-centered style and toward a higher level for student-centered style. For interactive styles, the ELE is more broadly distributed from low to high levels. The results suggest that interactive teaching may be influenced by a more diverse set of factors. Since the interactive style involves interactions among multiple participants (teachers, students, and groups) and is characterized by interactive cycles throughout, the stimulation of ELE in the interactive style may depend not only on the form of the activities, but also on the design of the interaction processes. For example, the design of question chains and the way of guidance are both important factors in promoting ELE. However, interactions that do not match students' proficiency levels may be counter-productive to ELE. This suggests that teachers need to pay particular attention to the design of interactive teaching to help all students develop their desirable learning pathways.</p> <p>In addition, the results also demonstrated that MP-FERS can produce accurate real-time measurements of ELE at a fine-grained temporal resolution, echoing the current literature on measuring emotion in the teaching process (Liaw et al., [<reflink idref="bib52" id="ref142">52</reflink>]). This feature makes it possible to study the temporal variation in ELE levels in response to different teaching activities and can help teachers improve classroom instruction in places where students' ELE is low. We analyzed the temporal variation in MP-FERS-measured ELE in one lesson. The synchronous changes in ELE among high- and low-performing groups across the timeline reflect their immediate responses to teaching activities, demonstrating the sensitivity of ELE to teaching variations. The results also reveal differences in the ELE responses between the high-performing and low-performing groups. In the student-centered style, the difference in ELE is small. Student-centered teaching is more open-ended and students have a greater sense of self-control, so low-performing groups may be emotionally satisfied through a variety of activities. In contrast, in most of the teacher-centered styles, the difference between the two groups is significant and even showed an opposite trend (minutes 15–20). In a teacher-centered style characterized by lecturing, students' emotional satisfaction may be more related to the ability to keep up with the teacher's lectures. This suggests that teachers need to pay attention to the difficulty of lecturing content and develop desirable learning pathways that can promote comprehension among low-performing students. On the other hand, teachers can also learn from teaching formats that produce high engagement for all students and thus tailor their future instruction toward a higher level of learning engagement.</p> <p>Notably, facial expressions may be subject to cultural variations, and ELE may vary across classroom settings, disciplines, and demographic groups. Follow-up studies should expand into other science disciplines and student populations. In addition, further research could consider digging deeper into the value-added effects of students' emotions on learning performance, such as retention of scientific concepts and participation in science activities, enabling educators to recognize how fostering ELE can contribute to learning outcomes in the science classroom.</p> <p>The application of artificial intelligence (AI) tools such as the MP-FERS has the potential to enhance the measurement of teaching effectiveness. However, variations in facial expressions across cultures and the privacy and ethical acceptance of AI technology should be carefully considered (Wu et al., [<reflink idref="bib106" id="ref143">106</reflink>]). For example, incomplete or biased data collection may lead to biased educational decisions. Overreliance on technology may reduce emotional communication between teachers and students and weaken teachers' discriminative ability. In conclusion, it is vital to respect each student's unique learning process and refrain from imposing uniform standards. Data derived from intelligent tools should inform and help refine teachers' pedagogical approaches and strategies.</p> <hd id="AN0183076991-28">Author Contribution</hd> <p>Conceptualization: Xiaoyu Tang, Lei Bao. Methodology: Yang Xiao, Xiaoyu Tang. Formal analysis and investigation: Yayun Gong. Writing—original draft preparation: Xiaoyu Tang, Yayun Gong. Writing—review and editing: Lei Bao, Yang Xiao. Funding acquisition: Xiaoyu Tang. Supervision: Lei Bao, Jianwen Xiong.</p> <hd id="AN0183076991-29">Funding</hd> <p>This work was supported in part by the National Social Science Foundation of China under Grant No. CHA200261. Any opinions expressed in this work are those of the authors and do not necessarily represent those of the funding agencies.</p> <hd id="AN0183076991-30">Data Availability</hd> <p>The models used herein uses a large-scale open dataset from the Internet open source, publicly available on Google Search. Data sheets are available upon request through the corresponding author.</p> <hd id="AN0183076991-31">Code Availability</hd> <p>Code is original and produced by the authors. Contact corresponding author to inquire about availability of code.</p> <hd id="AN0183076991-32">Data Availability</hd> <p>The data generated during the current study are partly available from the corresponding author on reasonable request. Because the class video data included images of students, we cannot share them with the readers due to the ethical reason.</p> <hd id="AN0183076991-33">Declarations</hd> <p></p> <hd id="AN0183076991-34">Ethics Approval</hd> <p>All human trials in this research meet the ethical standards of the Chinese Association for Ethical Research(CAES). This research is conducted with approval from the authors' institution.</p> <hd id="AN0183076991-35">Consent to Participate</hd> <p>Informed consent was obtained from all individual participants included in the study and their legal guardians.</p> <hd id="AN0183076991-36">Consent for Publication</hd> <p>The participants have provided informed consent for publication of their learning data in this article, and consented to the submission of the case report to the journal.</p> <hd id="AN0183076991-37">Conflict of Interest</hd> <p>The authors declare no competing interests.</p> <hd id="AN0183076991-38">Supplementary Information</hd> <p>Below is the link to the electronic supplementary material.</p> <p>Graph: Supplementary file1 (DOCX 242 KB)</p> <hd id="AN0183076991-39">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0183076991-40"> <title> References </title> <blist> <bibl id="bib1" idref="ref7" type="bt">1</bibl> <bibtext> Alimoglu MK, Yardim S, Uysal H. The effectiveness of TBL with real patients in neurology education in terms of knowledge retention, in-class engagement, and learner reactions. Advances in Physiology Education. 2017; 41; 1: 38-43. 10.1152/advan.00130.2016</bibtext> </blist> <blist> <bibl id="bib2" idref="ref9" type="bt">2</bibl> <bibtext> Altuwairqi K, Jarraya SK, Allinjawi A, Hammami M. A new emotion-based affective model to detect student's engagement. Journal of King Saud University-Computer and Information Sciences. 2021; 33; 1: 99-109. 10.1016/j.jksuci.2018.12.008</bibtext> </blist> <blist> <bibl id="bib3" idref="ref11" type="bt">3</bibl> <bibtext> Anthony C, Nicolas M. The recognition of emotions beyond facial expressions: Comparing emoticons specifically designed to convey basic emotions with other modes of expression. Computers in Human Behavior. 2021; 118: 106689. 10.1016/j.chb.2021.106689</bibtext> </blist> <blist> <bibl id="bib4" idref="ref74" type="bt">4</bibl> <bibtext> Areepattamannil S. Effects of inquiry-based science instruction on science achievement and interest in science: Evidence from Qatar. Journal of Educational Research. 2012; 105; 2: 134-146. 10.1080/00220671.2010.533717</bibtext> </blist> <blist> <bibl id="bib5" idref="ref109" type="bt">5</bibl> <bibtext> Arent SBartlett R, Gratton C, Rolf CG. Thayer's model of arousal and activation. Encyclopedia of International Sport Studies. 2005: London; Routledge</bibtext> </blist> <blist> <bibl id="bib6" idref="ref139" type="bt">6</bibl> <bibtext> Ashwin TS, Guddeti R. Affective database for e-learning and classroom environments using Indian students' faces, hand gestures and body postures. Future Generation Computer Systems. 2020; 108: 334-348. 10.1016/j.future.2020.02.075</bibtext> </blist> <blist> <bibl id="bib7" idref="ref140" type="bt">7</bibl> <bibtext> Baeten M, Kyndt E, Struyven K, Dochy F. Using student-centred learning environments to stimulate deep approaches to learning: Factors encouraging or discouraging their effectiveness. Educational Research Review. 2010; 5; 3: 243-260. 10.1016/J.EDUREV.2010.06.001</bibtext> </blist> <blist> <bibl id="bib8" idref="ref78" type="bt">8</bibl> <bibtext> Bakker AB, Vergel AIS, Kuntze J. Student engagement and performance: A weekly diary study on the role of openness. Motivation And Emotion. 2015; 39; 1: 49-62. 10.1007/S11031-014-9422-5</bibtext> </blist> <blist> <bibl id="bib9" idref="ref71" type="bt">9</bibl> <bibtext> Battersby GL, Beeley C, Baguley DA, Barker HD, Broad HD, Carey NC, Williams DP. Go Fischer: An Introductory Organic Chemistry Card Game. Journal of Chemical Education. 2020; 97; 8: 2226-2230. 10.1021/acs.jchemed.0c00504</bibtext> </blist> <blist> <bibtext> Ben-Eliyahu A, Moore D, Dorph R, Schunn CD. Investigating the multidimensionality of engagement: Affective, behavioral, and cognitive engagement across science activities and contexts. Contemporary Educational Psychology. 2018; 53: 87-105. 10.1016/J.CEDPSYCH.2018.01.002</bibtext> </blist> <blist> <bibtext> Blomeke S, Jentsch A, Ross N, Kaiser G, Konig J. Opening up the black box: Teacher competence, instructional quality, and students? learning progress. Learning and Instruction. 2022; 79: 101600. 10.1016/j.learninstruc.2022.101600</bibtext> </blist> <blist> <bibtext> Byusa, E, Kampire, E, & Mwesigye, A. R. (2022). Game-based learning approach on students? motivation and understanding of chemistry concepts: A systematic review of literature. Heliyon, 8(5). https://doi.org/10.1016/j.heliyon.2022.e09541</bibtext> </blist> <blist> <bibtext> Cao, J, Wang, H, Hu, P, & Miao, J. (2008). PAD Model Based Facial Expression Analysis. Paper presented at the Advances in Visual Computing, Berlin, Heidelberg.</bibtext> </blist> <blist> <bibtext> Capar G, Tarim K. Efficacy of the cooperative learning method on mathematics achievement and attitude: A meta-analysis research. Educational Sciences-Theory & Practice. 2015; 15; 2: 553-559</bibtext> </blist> <blist> <bibtext> Chen D, Wen G, Li H, Chen R, Li C. Multi-relations aware network for in-the-wild facial expression recognition. Ieee Transactions on Circuits and Systems for Video Technology. 2023; 33; 8: 3848-59. 10.1109/TCSVT.2023.3234312</bibtext> </blist> <blist> <bibtext> Chen, Y, Bosch, N, & 'Mello, S. D. (2015). Video-Based Affect Detection in Noninteractive Learning Environments. Paper presented at the 8th International Conference on Educational Data Mining. Madrid, Spain</bibtext> </blist> <blist> <bibtext> Chi MTH, Wylie R. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist. 2014; 49; 4: 219-243. 10.1080/00461520.2014.965823</bibtext> </blist> <blist> <bibtext> China, T. M. o. E. o. t. P. s. R. o. (2022). Science Curriculum Standards for compulsory Education: Beijing Normal University Publishing House</bibtext> </blist> <blist> <bibtext> Cohen, L, Manion, L, & Morrison, K. (2007). Research Methods in Education: Research Methods in Education.</bibtext> </blist> <blist> <bibtext> Connell, J. P, & Wellborn, J. G. (1991). Competence, autonomy, and relatedness: A motivational analysis of self-system processes. Journal of Personality Social Psychology, 65.</bibtext> </blist> <blist> <bibtext> Creswell, J. W, & Clark, V. L. (2011). Designing and Conducting Mixed Methods Research. Thousand Oaks</bibtext> </blist> <blist> <bibtext> D'Mello S, Graesser A. Dynamics of affective states during complex learning. Learning and Instruction. 2012; 22; 2: 145-157. 10.1016/J.LEARNINSTRUC.2011.10.001</bibtext> </blist> <blist> <bibtext> Delahunty T, Seery N, Lynch R. Exploring the use of electroencephalography to gather objective evidence of cognitive processing during problem solving. Journal of Science Education and Technology. 2018; 27; 2: 114-130. 10.1007/s10956-017-9712-2</bibtext> </blist> <blist> <bibtext> Dong J, Ke X. Using S-T teaching analysis to evaluate teacher-student interaction behavior. Biology Teaching. 2015; 40; 06: 11-12</bibtext> </blist> <blist> <bibtext> Dosovitskiy, A, Beyer, L, Kolesnikov, A, Weissenborn, D, Xiaohua, Z, Unterthiner, T,.. Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale</bibtext> </blist> <blist> <bibtext> Elias B, Kimberly TN, Roberta Michnick G, Kathy H-P. Investigating the contributions of active, playful learning to student interest and educational outcomes. Acta Psychologica. 2023; 238: 103983. 10.1016/j.actpsy.2023.103983</bibtext> </blist> <blist> <bibtext> Engels MC, Spilt JL, Denies K, Verschueren K. The role of affective teacher-student relationships in adolescents' school engagement and achievement trajectories. Learning and Instruction. 2021; 75: 101485. 10.1016/J.LEARNINSTRUC.2021.101485</bibtext> </blist> <blist> <bibtext> Eva T, Kathleen M. Teaching routines and student-centered mathematics instruction: The essential role of conferring to understand student thinking and reasoning. The Journal of Mathematical Behavior. 2023; 70: 101032. 10.1016/j.jmathb.2023.101032</bibtext> </blist> <blist> <bibtext> Fredricks JA, Blumenfeld PC, Paris AH. School engagement: Potential of the concept, state of the evidence. Review of Educational Research. 2004; 74; 1: 59-109. 10.3102/00346543074001059</bibtext> </blist> <blist> <bibtext> Fredricks JA, Hofkens T, Wang MT, Mortenson E, Scott P. Supporting girls' and boys' engagement in math and science learning: A mixed methods study. Journal of Research in Science Teaching. 2018; 55; 2: 271-298. 10.1002/tea.21419</bibtext> </blist> <blist> <bibtext> Freeman, A, Becker, S. A, & Cummins, M. (2017). NMC/CoSN Horizon Report: 2017 K-12 Edition. In.</bibtext> </blist> <blist> <bibtext> Fu D, Zhang H. Educational Information Processing. 2001; Beijing Normal University Publishing House</bibtext> </blist> <blist> <bibtext> Geertshuis SA. Slaves to our emotions: Examining the predictive relationship between emotional well-being and academic outcomes. Active Learning in Higher Education. 2019; 20; 2: 153-166. 10.1177/1469787418808932</bibtext> </blist> <blist> <bibtext> SW Gilroy M Cavazza M Benayoun 2009 Using affective trajectories to describe states of flow in interactive art Paper Presented at the Proceedings of the International Conference on Advances in Computer Entertainment Technology, Athens, Greece. https://doi.org/10.1145/1690388.1690416</bibtext> </blist> <blist> <bibtext> González-Romá V, Schaufeli WB, Bakker AB, Lloret S. Burnout and work engagement: Independent factors or opposite poles?. Journal of Vocational Behavior. 2006; 68; 1: 165-174. 10.1016/j.jvb.2005.01.003</bibtext> </blist> <blist> <bibtext> Greene BA. Measuring cognitive engagement with self-report scales: Reflections from over 20 years of research. Educational Psychologist. 2015; 50; 1: 14-30. 10.1080/00461520.2014.989230</bibtext> </blist> <blist> <bibtext> JC Greene 2007 Mixed Methods in Social Inquiry San Francisco</bibtext> </blist> <blist> <bibtext> Guo Y, Sun S, Breit-Smith A, Morrison FJ, Connor CMD. Behavioral engagement and reading achievement in elementary-school-age children: A longitudinal cross-lagged analysis. Journal of Educational Psychology. 2014; 107; 2: 332-347. 10.1037/A0037638</bibtext> </blist> <blist> <bibtext> Hair, J. F, Black, W. C, Babin, B. J, & Anderson, R. E. (2010). Multivariate Data Analysis: A Global Perspective: Multivariate Data Analysis: A Global Perspective.</bibtext> </blist> <blist> <bibtext> Henrie CR, Halverson LR, Graham CR. Measuring student engagement in technology-mediated learning. Computer Education. 2015; 90; 90: 36-53. 10.1016/J.COMPEDU.2015.09.005</bibtext> </blist> <blist> <bibtext> Hernández MM, Eisenberg N, Valiente C, Vanschyndel SK, Spinrad TL, Silva KM, Thompson MS. Emotional expression in school context, social relationships, and academic adjustment in kindergarten. Emotion. 2016; 16; 4: 553. 10.1037/emo0000147</bibtext> </blist> <blist> <bibtext> Howe C, Hennessy S, Mercer N, Vrikki M, Wheatley L. Teacher-student dialogue during classroom teaching: Does it really impact on student outcomes?. Journal of the Learning Sciences. 2019; 28; 4–5: 462-512. 10.1080/10508406.2019.1573730</bibtext> </blist> <blist> <bibtext> Jia J, Wu Z, Zhang S, Meng HM, Cai L. Head and facial gestures synthesis using PAD model for an expressive talking avatar. Multimedia Tools and Applications. 2014; 73; 1: 439-461. 10.1007/s11042-013-1604-8</bibtext> </blist> <blist> <bibtext> Jin, R, Zhao, S, Hao, Z, Xu, Y, Xu, T, & Chen, E. (2022). AVT: Au-Assisted Visual Transformer for Facial Expression Recognition. Transformer for Facial Expression Recognition. Paper presented at the 2022 IEEE International Conference on Image Processing (ICIP). Bordeaux, France. https://doi.org/10.1109/ICIP46576.2022.9897960</bibtext> </blist> <blist> <bibtext> Joshi A, Bhaskar P. Qualitative study on critical traits of teacher for effective teaching in higher education institutions. International Journal of Learning and Change. 2022; 14; 4: 390-408. 10.1504/ijlc.2022.124466</bibtext> </blist> <blist> <bibtext> Kaiyue L, Zhong S, Min X. Artificial intelligent based video analysis on the teaching interaction patterns in classroom environment. International Journal of Information and Education Technology. 2021; 11; 3: 126-130. 10.18178/ijiet.2021.11.3.1500</bibtext> </blist> <blist> <bibtext> Kang J, Keinonen T. The effect of student-centered approaches on students' interest and achievement in science: Relevant topic-based, open and guided inquiry-based, and discussion-based approaches. Research in Science Education. 2018; 48; 4: 865-885. 10.1007/s11165-016-9590-2</bibtext> </blist> <blist> <bibtext> Kateřina L. Socialization of a student teacher on teaching practice into the discursive community of the classroom: Between a teacher-centered and a learner-centered approach. Learning, Culture and Social Interaction. 2019; 22: 100314. 10.1016/j.lcsi.2019.05.001</bibtext> </blist> <blist> <bibtext> Kennedy MM. How does professional development improve teaching?. Review of Educational Research. 2016; 86; 4: 945-980. 10.3102/0034654315626800</bibtext> </blist> <blist> <bibtext> Kuhbandner C, Pekrun R, Maier MA. The role of positive and negative affect in the omirroringo of other persons' actions. Cognition & Emotion. 2010; 24; 7: 1182-1190. 10.1080/02699930903119196</bibtext> </blist> <blist> <bibtext> Li X, Wu J, Huang S. Analysis of high school biology quality course based on improved S-T analysis method. Journal of Teaching and Management. 2021; 21: 3</bibtext> </blist> <blist> <bibtext> Liaw H, Yu YR, Chou CC, Chiu MH. Relationships between facial expressions, prior knowledge, and multiple representations: A case of conceptual change for kinematics instruction. Journal of Science Education and Technology. 2021; 30; 2: 227-238. 10.1007/S10956-020-09863-3</bibtext> </blist> <blist> <bibtext> Linnenbrink EASchutz PA, Pekrun R. Chapter 7 - The role of affect in student learning: A multi-dimensional approach to considering the interaction of affect, motivation, and engagement. Emotion in Education. 2007; Academic Press: 107-124. 10.1016/B978-012372545-5/50008-3</bibtext> </blist> <blist> <bibtext> Liu H, Cai H, Li Q, Li X, Xiao H. Adaptive multilayer perceptual attention network for facial expression recognition. Ieee Transactions on Circuits and Systems for Video Technology. 2022; 32; 9: 6253-6266. 10.1109/tcsvt.2022.3165321</bibtext> </blist> <blist> <bibtext> Liu, L, Du, W, Wang, P, & Jing, M. (2014). Improvement of S-T analysis method and analysis of national high school chemistry quality lessons. Education in Chemistry (07) 19–22</bibtext> </blist> <blist> <bibtext> Liu S, Liu S, Liu Z, Peng X, Yang Z. Automated detection of emotional and cognitive engagement in MOOC discussions to predict learning achievement. Computers & Education. 2022; 181: 104461. 10.1016/j.compedu.2022.104461</bibtext> </blist> <blist> <bibtext> Liu T, Wang J, Yang B, Wang X. Facial expression recognition method with multi-label distribution learning for non-verbal behavior understanding in the classroom. Infrared Physics & Technology. 2021; 112: 103594. 10.1016/J.INFRARED.2020.103594</bibtext> </blist> <blist> <bibtext> Liu Y, Tao L, Fu X. The analysis of PAD emotional state model based on emotion pictures. Journal of Image and Graphics. 2009; 14; 05: 753-758</bibtext> </blist> <blist> <bibtext> Martínez-Sierra GG. High school students' emotional experiences in mathematics classes. Research in Mathematics Education. 2014; 16; 3: 17. 10.1080/14794802.2014.895676</bibtext> </blist> <blist> <bibtext> Mehrabian A. Framework for a comprehensive description and measurement of emotional states. Genetic Social and General Psychology Monographs. 1995; 121; 3: 339-361</bibtext> </blist> <blist> <bibtext> Mehrabian A. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology. 1996; 14; 4: 261-292. 10.1007/BF02686918</bibtext> </blist> <blist> <bibtext> Mehrabian, A, & Russell, J. A. (1974). An approach to environmental psychology. MIT Press</bibtext> </blist> <blist> <bibtext> Meinhardt J, Pekrun R. Attentional resource allocation to emotional events: An ERP study. Cognition and Emotion. 2003; 17; 3: 477-500. 10.1080/02699930244000039</bibtext> </blist> <blist> <bibtext> Mikeska JN, Holtzman S, McCaffrey DF, Liu S, Shattuck T. Using classroom observations to evaluate science teaching: Implications of lesson sampling for measuring science teaching effectiveness across lesson types. Science Education. 2019; 103; 1: 123-144. 10.1002/SCE.21482</bibtext> </blist> <blist> <bibtext> Mollahosseini, A, Chan, D, & Mahoor, M. H. (2016). Going deeper in facial expression recognition using deep neural networks. Paper presented at the Workshop on Applications of Computer Vision.</bibtext> </blist> <blist> <bibtext> Muenks K, Wigfield A, Yang JS, O'Neal CR. How true is grit? Assessing its relations to high school and college students' personality characteristics, self-regulation, engagement, and achievement. Journal of Educational Psychology. 2017; 109; 5: 599-620. 10.1037/EDU0000153</bibtext> </blist> <blist> <bibtext> Muñoz-García A, Villena-Martínez MD. Influences of learning approaches, student engagement, and satisfaction with learning on measures of sustainable behavior in a social sciences student sample. Sustainability. 2021; 13; 2: 541. 10.3390/SU13020541</bibtext> </blist> <blist> <bibtext> Nakamura, J, & Csikszentmihalyi, M. (2014). The concept of flow. Flow and the Foundations of Positive Psychology 25.</bibtext> </blist> <blist> <bibtext> Newmann, F. M. (1992). Student engagement and achievement in American secondary schools.</bibtext> </blist> <blist> <bibtext> North, B, Diab, M, Lameras, P, Zaraik, J, & Fischer, H. (2021). Developing a Platform for using Game-Based Learning in Vocational Education and Training. Paper presented at the 2021 IEEE Global Engineering Education Conference (EDUCON).</bibtext> </blist> <blist> <bibtext> Office for Standards in Education, C. s. S. a. S. O. (2010). The Evaluation Schedule for Schools. In.</bibtext> </blist> <blist> <bibtext> Olivier E, Galand B, Morin AJS, Hospel V. Need-supportive teaching and student engagement in the classroom: Comparing the additive, synergistic, and global contributions. Learning And Instruction. 2021; 71: 101389. 10.1016/J.LEARNINSTRUC.2020.101389</bibtext> </blist> <blist> <bibtext> Ontario, T. M. (2014). Achieving excellence: A renewed vision for education in Ontario. Government of Ontario.</bibtext> </blist> <blist> <bibtext> Park S, Holloway SD, Arendtsz A, Bempechat J, Li J. What makes students engaged in learning? A time-use study of within- and between-individual predictors of emotional engagement in low-performing high schools. Journal of Youth and Adolescence. 2012; 41; 3: 390-401. 10.1007/S10964-011-9738-3</bibtext> </blist> <blist> <bibtext> Pei J, Shan P. A micro-expression recognition algorithm for students in classroom learning based on convolutional neural network. Traitement Du Signal. 2019; 36; 6: 557-563. 10.18280/TS.360611</bibtext> </blist> <blist> <bibtext> Pekrun R. The control-value theory of achievement emotions: Assumptions, corollaries, and implications for educational research and practice. Educational Psychology Review. 2006; 18; 4: 315-341. 10.1007/s10648-006-9029-9</bibtext> </blist> <blist> <bibtext> Pekrun R, Elliot AJ, Maier MA. Achievement goals and achievement emotions: Testing a model of their joint relations with academic performance. Journal of Educational Psychology. 2009; 101; 1: 115-135. 10.1037/A0013383</bibtext> </blist> <blist> <bibtext> Pekrun R, Goetz T, Titz W, Perry RP. Academic emotions in students' self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist. 2002; 37; 2: 91-105. 10.1207/S15326985EP3702_4</bibtext> </blist> <blist> <bibtext> Pekrun R, Lichtenfeld S, Marsh HW, Murayama K, Goetz T. Achievement emotions and academic performance: Longitudinal models of reciprocal effects. Child Development. 2017; 88; 5: 1653-1670. 10.1111/cdev.12704</bibtext> </blist> <blist> <bibtext> Pekrun R, Linnenbrink-Garcia LChristenson SL, Reschly AL, Wylie C. Academic Emotions and Student Engagement. Handbook of Research on Student Engagement. 2012: US; Springer: 259-282. 10.1007/978-1-4614-2018-7_12</bibtext> </blist> <blist> <bibtext> Prescod DJ, Daire AP, Young C, Dagley M, Georgiopoulos M. Exploring negative career thoughts between STEM-declared and STEM-interested students. Journal of Employment Counseling. 2018; 55; 4: 166-175. 10.1002/joec.12096</bibtext> </blist> <blist> <bibtext> Putwain DW, Becker S, Symes W, Pekrun R. Reciprocal relations between students' academic enjoyment, boredom, and achievement over time. Learning and Instruction. 2018; 54: 73-81. 10.1016/j.learninstruc.2017.08.004</bibtext> </blist> <blist> <bibtext> Putwain DW, Wood P, Pekrun R. Achievement emotions and academic achievement: Reciprocal relations and the moderating influence of academic buoyancy. Journal of Educational Psychology. 2022; 114; 1: 108-126. 10.1037/edu0000637</bibtext> </blist> <blist> <bibtext> Reinhold F, Strohmaier A, Hoch S, Reiss K, Böheim R, Seidel T. Process data from electronic textbooks indicate students' classroom engagement. Learning and Individual Differences. 2020; 83: 101934. 10.1016/J.LINDIF.2020.101934</bibtext> </blist> <blist> <bibtext> Renninger KA, Bachrach JE. Studying triggers for interest and engagement using observational methods. Educational Psychologist. 2015; 50; 1: 58-69. 10.1080/00461520.2014.999920</bibtext> </blist> <blist> <bibtext> Renninger, K. A, Hidi, S, & Krapp, A. (1994). The role of interest in learning and development. American Journal of Psychology, 107(2) https://doi.org/10.4324/9781315807430</bibtext> </blist> <blist> <bibtext> Sadler PM, Sonnert G, Hazari Z, Tai R. Stability and volatility of STEM career interest in high school: A gender study. Science Education. 2012; 96; 3: 411-427. 10.1002/sce.21007</bibtext> </blist> <blist> <bibtext> Schelfhout S, Wille B, Fonteyne L, Roels E, De Fruyt F, Duyck W. From interest assessment to study orientation: An empirical advice set engine. Journal of Experimental Education. 2021; 89; 1: 169-195. 10.1080/00220973.2019.1605327</bibtext> </blist> <blist> <bibtext> Schukajlow S, Rakoczy K. The power of emotions: Can enjoyment and boredom explain the impact of individual preconditions and teaching methods on interest and performance in mathematics?. Learning and Instruction. 2016; 44: 117-127. 10.1016/j.learninstruc.2016.05.001</bibtext> </blist> <blist> <bibtext> Sedova K, Sedlacek M, Svaricek R, Majcik M, Navratilova J, Drexlerova A, Salamounova Z. Do those who talk more learn more? The relationship between student classroom talk and student achievement. Learning and Instruction. 2019; 63: 101217. 10.1016/J.LEARNINSTRUC.2019.101217</bibtext> </blist> <blist> <bibtext> Shen J, Yang H, Li J, Cheng Z. Assessing learning engagement based on facial expression recognition in MOOC's scenario. Multimedia Systems. 2021. 10.1007/s00530-021-00854-x</bibtext> </blist> <blist> <bibtext> Effect of cooperative learning on chemistry students' achievement in Rwandan Day-upper secondary schools. (2021). European Journal of Educational Research,10(4), 10.</bibtext> </blist> <blist> <bibtext> Skinner EA, Belmont MJ. Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement across the school year. Journal of Educational Psychology. 1993; 85; 4: 571-581. 10.1037/0022-0663.85.4.571</bibtext> </blist> <blist> <bibtext> Sun B, Liu YN, Chen JB, Luo JH, Zhang D. Emotion analysis based on facial expression recognition in smart learning environment. Modern Distance Education Research. 2015; 2; 8: 96-103</bibtext> </blist> <blist> <bibtext> Sun H-L, Sun T, Sha F-Y, Gu X-Y, Hou X-R, Zhu F-Y, Fang P-T. The influence of teacher-student interaction on the effects of online learning: Based on a serial mediating model. Frontiers in Psychology. 2022; 13: 779217. 10.3389/fpsyg.2022.779217</bibtext> </blist> <blist> <bibtext> Taub M, Sawyer R, Smith A, Rowe J, Azevedo R, Lester J. The agency effect: The impact of student agency on learning, emotions, and problem-solving behaviors in a game-based learning environment. Computers & Education. 2020; 147: 103781. 10.1016/j.compedu.2019.103781</bibtext> </blist> <blist> <bibtext> Trobst S, Kleickmann T, Lange-Schubert K, Rothkopf A, Moller K. Instruction and students' declining interest in science: An analysis of German Fourth- and Sixth-Grade classrooms. American Educational Research Journal. 2016; 53; 1: 162-193. 10.3102/0002831215618662</bibtext> </blist> <blist> <bibtext> Tsai JL, Blevins E, Bencharit LZ, Chim L, Fung HH, Yeung DY. Cultural variation in social judgments of smiles: The role of ideal affect. Journal of Personality and Social Psychology. 2019; 116; 6: 966-988. 10.1037/pspp0000192</bibtext> </blist> <blist> <bibtext> Vanneste P, Oramas J, Verelst T, Tuytelaars T, Raes A, Depaepe F, Van den Noortgate W. Computer vision and human behaviour, emotion and cognition detection: A use case on student engagement. Mathematics. 2021; 9; 3: 287. 10.3390/math9030287https://<ulink href="http://www.mdpi.com/2227-7390/9/3/287">www.mdpi.com/2227-7390/9/3/287</ulink></bibtext> </blist> <blist> <bibtext> Wang, J, Ma, X, Sun, J, Zhao, Z, & Zhu, Y. (2014). Puzzlement detection from facial expression using active appearance models and support vector machines. International Journal of Signal Processing Image Processing Pattern Recognition 7https://doi.org/10.14257/ijsip.2014.7.5.30</bibtext> </blist> <blist> <bibtext> Wang, K, Peng, X, Yang, J, Lu, S, Qiao, Y, & Ieee. (2020). Suppressing Uncertainties for Large-Scale Facial Expression Recognition. Paper presented at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Electr Network.</bibtext> </blist> <blist> <bibtext> Wang M-T, Eccles JS. Adolescent behavioral, emotional, and cognitive engagement trajectories in school and their differential relations to educational success. Journal of Research on Adolescence. 2012; 22; 1: 31-39. 10.1111/J.1532-7795.2011.00753.X</bibtext> </blist> <blist> <bibtext> Wang M-T, Willett JB, Eccles JS. The assessment of school engagement: Examining dimensionality and measurement invariance by gender and race/ethnicity. Journal of School Psychology. 2011; 49; 4: 465-480. 10.1016/J.JSP.2011.04.001</bibtext> </blist> <blist> <bibtext> Watson WR, Watson SL, Magar ST, Tay L. Comparing attitudinal learning of large enrolment active learning and lecture classes. Innovations in Education and Teaching International. 2021; 58; 2: 146-156. 10.1080/14703297.2019.1711440</bibtext> </blist> <blist> <bibtext> Whitehill J, Serpell Z, Lin YC, Foster A, Movellan JR. The faces of engagement: Automatic recognition of student engagement from facial expressions. IEEE Transactions on Affective Computing. 2014; 5; 1: 86-98. 10.1109/TAFFC.2014.2316163</bibtext> </blist> <blist> <bibtext> Wu H, Tu Y, Tan L. Education risks and its avoidance in the era of artificial intelligence. Modern Educational Technology. 2020; 030; 004: 18-24</bibtext> </blist> <blist> <bibtext> Wundt, W. (1980). Outlines of Psychology: US:Springer.</bibtext> </blist> <blist> <bibtext> Xintong L, Christi B, Amanda AO. Positive teacher-student relationships may lead to better teaching. Learning and Instruction. 2022; 80: 101581. 10.1016/j.learninstruc.2022.101581</bibtext> </blist> <blist> <bibtext> Xue, F, Wang, Q, & Guo, G. (2021). TransFER: Learning Relation-aware Facial Expression Representations with Transformers. Paper presented at the 18th IEEE/CVF International Conference on Computer Vision (ICCV), Electr Network.</bibtext> </blist> <blist> <bibtext> Yang X, Zhang M, Kong L, Wang Q, Hong JC. The effects of scientific self-efficacy and cognitive anxiety on science engagement with the "Question-Observation-ng-Explanation" model during school disruption in COVID-19 pandemic. Journal of Science Education and Technology. 2021; 30; 3: 1-14. 10.1007/S10956-020-09877-X</bibtext> </blist> <blist> <bibtext> Zhang, Y, Wang, C, Ling, X, & Deng, W. (2022). Learn from All: Erasing Attention Consistency for Noisy Label Facial Expression Recognition. Paper presented at the 17th European Conference on Computer Vision (ECCV), Tel Aviv, ISRAEL.</bibtext> </blist> <blist> <bibtext> Zhao Z, Liu Q, Wang S. Learning deep global multi-scale and local attention features for facial expression recognition in the wild. Ieee Transactions on Image Processing. 2021; 30: 6544-6556. 10.1109/tip.2021.3093397</bibtext> </blist> <blist> <bibtext> Zhu X, Chen Z. Dual-modality spatiotemporal feature learning for spontaneous facial expression recognition in e-learning using hybrid deep neural network. Visual Computer. 2019. 10.1007/s00371-019-01660-3</bibtext> </blist> </ref> <aug> <p>By Xiaoyu Tang; Yayun Gong; Yang Xiao; Jianwen Xiong and Lei Bao</p> <p>Reported by Author; Author; Author; Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib11" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib45" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib26" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib49" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib95" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib108" firstref="ref6"></nolink> <nolink nlid="nl7" bibid="bib48" firstref="ref8"></nolink> <nolink nlid="nl8" bibid="bib28" firstref="ref10"></nolink> <nolink nlid="nl9" bibid="bib42" firstref="ref12"></nolink> <nolink nlid="nl10" bibid="bib71" firstref="ref13"></nolink> <nolink nlid="nl11" bibid="bib73" firstref="ref14"></nolink> <nolink nlid="nl12" bibid="bib18" firstref="ref15"></nolink> <nolink nlid="nl13" bibid="bib94" firstref="ref16"></nolink> <nolink nlid="nl14" bibid="bib81" firstref="ref17"></nolink> <nolink nlid="nl15" bibid="bib87" firstref="ref18"></nolink> <nolink nlid="nl16" bibid="bib88" firstref="ref19"></nolink> <nolink nlid="nl17" bibid="bib64" firstref="ref20"></nolink> <nolink nlid="nl18" bibid="bib10" firstref="ref21"></nolink> <nolink nlid="nl19" bibid="bib57" firstref="ref22"></nolink> <nolink nlid="nl20" bibid="bib65" firstref="ref23"></nolink> <nolink nlid="nl21" bibid="bib27" firstref="ref24"></nolink> <nolink nlid="nl22" bibid="bib66" firstref="ref25"></nolink> <nolink nlid="nl23" bibid="bib29" firstref="ref26"></nolink> <nolink nlid="nl24" bibid="bib90" firstref="ref27"></nolink> <nolink nlid="nl25" bibid="bib103" firstref="ref28"></nolink> <nolink nlid="nl26" bibid="bib36" firstref="ref29"></nolink> <nolink nlid="nl27" bibid="bib20" firstref="ref30"></nolink> <nolink nlid="nl28" bibid="bib69" firstref="ref31"></nolink> <nolink nlid="nl29" bibid="bib72" firstref="ref32"></nolink> <nolink nlid="nl30" bibid="bib93" firstref="ref34"></nolink> <nolink nlid="nl31" bibid="bib68" firstref="ref36"></nolink> <nolink nlid="nl32" bibid="bib86" firstref="ref37"></nolink> <nolink nlid="nl33" bibid="bib53" firstref="ref38"></nolink> <nolink nlid="nl34" bibid="bib77" firstref="ref39"></nolink> <nolink nlid="nl35" bibid="bib22" firstref="ref40"></nolink> <nolink nlid="nl36" bibid="bib78" firstref="ref42"></nolink> <nolink nlid="nl37" bibid="bib80" firstref="ref44"></nolink> <nolink nlid="nl38" bibid="bib41" firstref="ref45"></nolink> <nolink nlid="nl39" bibid="bib79" firstref="ref47"></nolink> <nolink nlid="nl40" bibid="bib82" firstref="ref48"></nolink> <nolink nlid="nl41" bibid="bib83" firstref="ref49"></nolink> <nolink nlid="nl42" bibid="bib84" firstref="ref50"></nolink> <nolink nlid="nl43" bibid="bib33" firstref="ref51"></nolink> <nolink nlid="nl44" bibid="bib54" firstref="ref52"></nolink> <nolink nlid="nl45" bibid="bib56" firstref="ref53"></nolink> <nolink nlid="nl46" bibid="bib35" firstref="ref54"></nolink> <nolink nlid="nl47" bibid="bib102" firstref="ref55"></nolink> <nolink nlid="nl48" bibid="bib63" firstref="ref56"></nolink> <nolink nlid="nl49" bibid="bib76" firstref="ref57"></nolink> <nolink nlid="nl50" bibid="bib50" firstref="ref58"></nolink> <nolink nlid="nl51" bibid="bib110" firstref="ref59"></nolink> <nolink nlid="nl52" bibid="bib17" firstref="ref60"></nolink> <nolink nlid="nl53" bibid="bib47" firstref="ref62"></nolink> <nolink nlid="nl54" bibid="bib89" firstref="ref63"></nolink> <nolink nlid="nl55" bibid="bib85" firstref="ref65"></nolink> <nolink nlid="nl56" bibid="bib97" firstref="ref66"></nolink> <nolink nlid="nl57" bibid="bib92" firstref="ref67"></nolink> <nolink nlid="nl58" bibid="bib70" firstref="ref68"></nolink> <nolink nlid="nl59" bibid="bib96" firstref="ref69"></nolink> <nolink nlid="nl60" bibid="bib59" firstref="ref70"></nolink> <nolink nlid="nl61" bibid="bib12" firstref="ref72"></nolink> <nolink nlid="nl62" bibid="bib30" firstref="ref73"></nolink> <nolink nlid="nl63" bibid="bib14" firstref="ref75"></nolink> <nolink nlid="nl64" bibid="bib38" firstref="ref79"></nolink> <nolink nlid="nl65" bibid="bib74" firstref="ref82"></nolink> <nolink nlid="nl66" bibid="bib40" firstref="ref83"></nolink> <nolink nlid="nl67" bibid="bib31" firstref="ref84"></nolink> <nolink nlid="nl68" bibid="bib62" firstref="ref85"></nolink> <nolink nlid="nl69" bibid="bib98" firstref="ref86"></nolink> <nolink nlid="nl70" bibid="bib100" firstref="ref88"></nolink> <nolink nlid="nl71" bibid="bib52" firstref="ref89"></nolink> <nolink nlid="nl72" bibid="bib16" firstref="ref90"></nolink> <nolink nlid="nl73" bibid="bib113" firstref="ref91"></nolink> <nolink nlid="nl74" bibid="bib75" firstref="ref92"></nolink> <nolink nlid="nl75" bibid="bib91" firstref="ref93"></nolink> <nolink nlid="nl76" bibid="bib99" firstref="ref94"></nolink> <nolink nlid="nl77" bibid="bib60" firstref="ref95"></nolink> <nolink nlid="nl78" bibid="bib61" firstref="ref96"></nolink> <nolink nlid="nl79" bibid="bib13" firstref="ref97"></nolink> <nolink nlid="nl80" bibid="bib43" firstref="ref98"></nolink> <nolink nlid="nl81" bibid="bib34" firstref="ref99"></nolink> <nolink nlid="nl82" bibid="bib21" firstref="ref100"></nolink> <nolink nlid="nl83" bibid="bib37" firstref="ref101"></nolink> <nolink nlid="nl84" bibid="bib25" firstref="ref105"></nolink> <nolink nlid="nl85" bibid="bib107" firstref="ref110"></nolink> <nolink nlid="nl86" bibid="bib58" firstref="ref112"></nolink> <nolink nlid="nl87" bibid="bib19" firstref="ref117"></nolink> <nolink nlid="nl88" bibid="bib23" firstref="ref119"></nolink> <nolink nlid="nl89" bibid="bib39" firstref="ref120"></nolink> <nolink nlid="nl90" bibid="bib46" firstref="ref122"></nolink> <nolink nlid="nl91" bibid="bib32" firstref="ref123"></nolink> <nolink nlid="nl92" bibid="bib51" firstref="ref124"></nolink> <nolink nlid="nl93" bibid="bib24" firstref="ref126"></nolink> <nolink nlid="nl94" bibid="bib55" firstref="ref127"></nolink> <nolink nlid="nl95" bibid="bib136" firstref="ref130"></nolink> <nolink nlid="nl96" bibid="bib696" firstref="ref134"></nolink> <nolink nlid="nl97" bibid="bib609" firstref="ref136"></nolink> <nolink nlid="nl98" bibid="bib67" firstref="ref137"></nolink> <nolink nlid="nl99" bibid="bib105" firstref="ref138"></nolink> <nolink nlid="nl100" bibid="bib104" firstref="ref141"></nolink> <nolink nlid="nl101" bibid="bib106" firstref="ref143"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1460783
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Facial Expression Recognition for Probing Students' Emotional Engagement in Science Learning
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Xiaoyu+Tang%22">Xiaoyu Tang</searchLink><br /><searchLink fieldCode="AR" term="%22Yayun+Gong%22">Yayun Gong</searchLink><br /><searchLink fieldCode="AR" term="%22Yang+Xiao%22">Yang Xiao</searchLink><br /><searchLink fieldCode="AR" term="%22Jianwen+Xiong%22">Jianwen Xiong</searchLink><br /><searchLink fieldCode="AR" term="%22Lei+Bao%22">Lei Bao</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0003-3348-4198">0000-0003-3348-4198</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Science+Education+and+Technology%22"><i>Journal of Science Education and Technology</i></searchLink>. 2025 34(1):13-30.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 18
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Physics%22">Physics</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Instruction%22">Science Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Nonverbal+Communication%22">Nonverbal Communication</searchLink><br /><searchLink fieldCode="DE" term="%22Science+Achievement%22">Science Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Emotional+Response%22">Emotional Response</searchLink><br /><searchLink fieldCode="DE" term="%22Affective+Measures%22">Affective Measures</searchLink><br /><searchLink fieldCode="DE" term="%22Arousal+Patterns%22">Arousal Patterns</searchLink><br /><searchLink fieldCode="DE" term="%22Predictive+Validity%22">Predictive Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Learner+Engagement%22">Learner Engagement</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Methods%22">Teaching Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Instructional+Effectiveness%22">Instructional Effectiveness</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s10956-024-10143-7
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1059-0145<br />1573-1839
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Student engagement in science classroom is an essential element for delivering effective instruction. However, the popular method for measuring students' emotional learning engagement (ELE) relies on self-reporting, which has been criticized for possible bias and lacking fine-grained time solution needed to track the effects of short-term learning interactions. Recent research suggests that students' facial expressions may serve as an external representation of their emotions in learning. Accordingly, this study proposes a machine learning method to efficiently measure students' ELE in real classroom. Specifically, a facial expression recognition system based on a multiscale perception network (MP-FERS) was developed by combining the pleasure-displeasure, arousal-nonarousal, and dominance-submissiveness (PAD) emotion models. Data were collected from videos of six physics lessons with 108 students. Meanwhile, students' academic records and self-reported learning engagement were also collected. The results show that students' ELE measured by MP-FERS was a significant predictor of academic achievement and a better indicator of true learning status than self-reported ELE. Furthermore, MP-FERS can provide fine-grained time resolution on tracking the changes in students' ELE in response to different teaching environments such as teacher-centered or student-centered classroom activities. The results of this study demonstrate the validity and utility of MP-FERS in studying students' emotional learning engagement.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1460783
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1460783
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s10956-024-10143-7
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 18
        StartPage: 13
    Subjects:
      – SubjectFull: Physics
        Type: general
      – SubjectFull: Science Instruction
        Type: general
      – SubjectFull: Nonverbal Communication
        Type: general
      – SubjectFull: Science Achievement
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Emotional Response
        Type: general
      – SubjectFull: Affective Measures
        Type: general
      – SubjectFull: Arousal Patterns
        Type: general
      – SubjectFull: Predictive Validity
        Type: general
      – SubjectFull: Learner Engagement
        Type: general
      – SubjectFull: Teaching Methods
        Type: general
      – SubjectFull: Instructional Effectiveness
        Type: general
    Titles:
      – TitleFull: Facial Expression Recognition for Probing Students' Emotional Engagement in Science Learning
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Xiaoyu Tang
      – PersonEntity:
          Name:
            NameFull: Yayun Gong
      – PersonEntity:
          Name:
            NameFull: Yang Xiao
      – PersonEntity:
          Name:
            NameFull: Jianwen Xiong
      – PersonEntity:
          Name:
            NameFull: Lei Bao
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 02
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1059-0145
            – Type: issn-electronic
              Value: 1573-1839
          Numbering:
            – Type: volume
              Value: 34
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Science Education and Technology
              Type: main
ResultId 1