View in EDS HTML Full Text PDF Full Text

Your Body Tells How You Engage in Collaboration: Machine-Detected Body Movements as Indicators of Engagement in Collaborative Math Knowledge Building

Saved in:
Bibliographic Details
Title: Your Body Tells How You Engage in Collaboration: Machine-Detected Body Movements as Indicators of Engagement in Collaborative Math Knowledge Building
Language: English
Authors: Hanall Sung (ORCID 0000-0001-9076-3762), Mitchell J. Nathan
Source: British Journal of Educational Technology. 2024 55(5):1950-1973.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 24
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Education Level: Elementary Secondary Education
Higher Education
Postsecondary Education
Descriptors: Cooperative Learning, Motion, Human Body, Learning Analytics, Learning Processes, Mathematics Education, Learner Engagement, Elementary Secondary Education, Preservice Teachers, Technology Uses in Education, Geometry, Gamification, Computer Simulation, Nonverbal Learning, Verbal Learning
DOI: 10.1111/bjet.13473
ISSN: 0007-1013
1467-8535
Abstract: Collaborative learning, driven by knowledge co-construction and meaning negotiation, is a pivotal aspect of educational contexts. While gesture's importance in conveying shared meaning is recognized, its role in collaborative group settings remains understudied. This gap hinders accurate and equitable assessment and instruction, particularly for linguistically diverse students. Advancements in multimodal learning analytics, leveraging sensor technologies, offer innovative solutions for capturing and analysing body movements. This study employs these novel approaches to demonstrate how learners' machine-detected body movements during the learning process relate to their verbal and nonverbal contributions to the co-construction of embodied math knowledge. These findings substantiate the feasibility of utilizing learners' machine-detected body movements as a valid indicator for inferring their engagement with the collaborative knowledge construction process. In addition, we empirically validate that these inferred different levels of learner engagement indeed impact the desired learning outcomes of the intervention. This study contributes to our scientific understanding of multimodal approaches to knowledge expression and assessment in learning, teaching, and collaboration.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1434977
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGtNRymasHxp6H6qz511CZ9AAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDJD7Sf2hdLCJ3vrRWgIBEICBm6QmHW3AkOxlEcHyvnMxCYMCon-jit0D3VgJLMG4eIL2OyfjZjkzUT_lyXLnsK-fodTTF_WV6xykQGqbJ20x_b6SDDyb75vz_D-z-rA3w2QdinurpM4x69S9qc94crURoc_rEcEQs4jQ_yggHSsbCcE9o7feoeuJPhIaytgMkubJH_7hsb0UZOlHfHAUGaeBLj8ozd26QASOe8co
Text:
  Availability: 1
  Value: <anid>AN0178994769;58i01sep.24;2024Aug16.05:30;v2.2.500</anid> <title id="AN0178994769-1">Your body tells how you engage in collaboration: Machine‐detected body movements as indicators of engagement in collaborative math knowledge building </title> <p>Collaborative learning, driven by knowledge co‐construction and meaning negotiation, is a pivotal aspect of educational contexts. While gesture's importance in conveying shared meaning is recognized, its role in collaborative group settings remains understudied. This gap hinders accurate and equitable assessment and instruction, particularly for linguistically diverse students. Advancements in multimodal learning analytics, leveraging sensor technologies, offer innovative solutions for capturing and analysing body movements. This study employs these novel approaches to demonstrate how learners' machine‐detected body movements during the learning process relate to their verbal and nonverbal contributions to the co‐construction of embodied math knowledge. These findings substantiate the feasibility of utilizing learners' machine‐detected body movements as a valid indicator for inferring their engagement with the collaborative knowledge construction process. In addition, we empirically validate that these inferred different levels of learner engagement indeed impact the desired learning outcomes of the intervention. This study contributes to our scientific understanding of multimodal approaches to knowledge expression and assessment in learning, teaching, and collaboration.Practitioner notesWhat is already known about this topicPrevious research emphasizes the importance of gestures as essential tools for constructing common ground and reflecting shared meaning‐making in learning and teaching contexts.The prior studies in multimodal learning analytics (MMLA) suggest that certain forms of body movements and postures can be differentiated based on the automatic detection of upper body joint locations.Empirical observations indicate that co‐thought gestures typically involve smaller hand or arms movement that are closer to the gesturer's body than co‐speech gestures used in interpersonal communication.What this paper addsThis paper fills the research gap by examining the use of gestures in collaborative learning, offering insights into how individuals contribute verbally and nonverbally to collaborative knowledge construction.This paper introduces the concept of using machine‐detected body movements as a viable proxy for inferring learners' engagement in collaborative knowledge‐building activities.Leverages sensor technologies for automatic detection of body movements, the innovative approach in this work seeks to overcome the time‐intensive and laborious process of manually coding gestures.Implications for practice and/or policyBy recognizing the potential significance of learners' body movements in indicating engagement levels with collaborative knowledge‐building activities, instructors can set up computer‐supported collaborative learning (CSCL) environments to enable capturing these movements.Given the crucial role of gestures in learning, teaching, and collaboration, educators can create more equitable formative assessment practices for linguistically diverse students by developing strategies that align with multimodal forms of knowledge expression.Research can expand beyond mathematics to explore the transferability of these findings to other subjects, helping educators create comprehensive pedagogical approaches that leverage multimodal interactions across disciplines.</p> <p>Keywords: body movement; collaborative learning; embodied learning; engagement; gesture; multimodal learning analytics; sensor</p> <hd id="AN0178994769-2">INTRODUCTION</hd> <p>In collaborative learning, people share and negotiate meaning while co‐constructing knowledge (Suthers, [<reflink idref="bib32" id="ref1">32</reflink>]; Vygotsky, [<reflink idref="bib33" id="ref2">33</reflink>]; Wertsch, [<reflink idref="bib36" id="ref3">36</reflink>]). Extensive educational research underscores <emph>gesture</emph> as a crucial aspect of meaning‐making in learning and teaching (eg, Alibali et al., [<reflink idref="bib3" id="ref4">3</reflink>]; Alibali & Nathan, [<reflink idref="bib2" id="ref5">2</reflink>]; Walkington et al., [<reflink idref="bib34" id="ref6">34</reflink>]). However, much of this research has focused on individual rather than group settings, resulting in insufficient attention to gestures as a tool for collaboration and shared cognitive process. This research gap poses challenges in assessment and instruction, potentially underestimating students' nonverbal ways of engagement in collaborative knowledge building, especially among students from linguistically diverse backgrounds (Lee & Fradd, [<reflink idref="bib19" id="ref7">19</reflink>]). Examining gesture use in collaborative learning contexts will offer a more comprehensive understanding of verbal and nonverbal contributions to knowledge co‐construction.</p> <p>However, capturing and coding gestures within collaborative learning contexts has been acknowledged as demanding due to the reliance on human coders to annotate gestures by repeatedly viewing video recordings. Although this approach yields rich and contextualized information about meaning‐making through verbal and nonverbal expressions of knowledge, the manual coding process is time‐consuming and labour‐intensive. Recently, spurred by rapid advancements in sensor technologies, a nascent community of scholars in <emph>multimodal learning analytics</emph> (MMLA; Blikstein & Worsley, [<reflink idref="bib4" id="ref8">4</reflink>]) has taken on these challenges. MMLA researchers adopt novel methods to collect and analyse body movements (or gestures) with less human labour, leveraging sensors for automatic movement detection (eg, Echeverría et al., [<reflink idref="bib11" id="ref9">11</reflink>]; Ochoa et al., [<reflink idref="bib23" id="ref10">23</reflink>]; Schneider & Blikstein, [<reflink idref="bib24" id="ref11">24</reflink>]). Still, given the evolving nature of the MMLA field, this analytic approach necessitates further empirical investigations to establish connections between the <emph>meaning</emph> embedded in multimodal traces (in this context, body movements) and learning behaviours or constructs (in this context, engagement).</p> <p>In this study, we implemented an online action‐based learning intervention for K‐12 pre‐service math teachers (ie, college students majoring in math education) that enabled them to experience and reflect embodied mathematical thinking and reasoning. During the intervention, learners, referred to as pre‐service teachers (PSTs), engaged in a co‐design activity where they collaboratively discussed and developed a series of actions aimed at fostering students' embodied math knowledge in geometry classrooms. We posit that it is possible to infer learners' engagement in the co‐construction of embodied math knowledge using only a small amount of information: the automatic detection of variances in their body movements throughout the learning process. To validate the feasibility of using these relatively sparse data as a valid indicator of learners' engagement in the co‐design activity, we explore how learners' machine‐detected body movements relate to their verbal and nonverbal contributions to collaborative knowledge building. Further, our investigation aims to provide empirical evidence that the discerned variations in learner engagement levels within the intervention can potentially impact desired outcomes. Specifically, we explore how these differences influence PSTs' ability to interpret multimodal forms of knowledge expression during formative assessment practices.</p> <hd id="AN0178994769-3">THEORETICAL BACKGROUND</hd> <p></p> <hd id="AN0178994769-4">Gesture during collaboration: Co‐speech gesture and co‐thought gesture</hd> <p>Previous educational research emphasizes the crucial role of <emph>gesture</emph> in building common ground and reflecting shared meaning‐making in the contexts of learning and teaching mathematics (Alibali et al., [<reflink idref="bib3" id="ref12">3</reflink>]; Alibali & Nathan, [<reflink idref="bib2" id="ref13">2</reflink>]; Walkington et al., [<reflink idref="bib34" id="ref14">34</reflink>]). People frequently employ hand gestures and whole‐body movements in conjunction with their speech when sharing mathematics ideas (eg, Alibali & Nathan, [<reflink idref="bib1" id="ref15">1</reflink>], [<reflink idref="bib2" id="ref16">2</reflink>]; Davidsen & Ryberg, [<reflink idref="bib10" id="ref17">10</reflink>]; Edwards et al., [<reflink idref="bib12" id="ref18">12</reflink>]). Many multimodal analysis studies, aimed at identifying gestures linked to meaning‐making, focus on <emph>representational gestures</emph> that "depict action, motion, or shape, or that indicate location or trajectory" (Kita et al., [<reflink idref="bib18" id="ref19">18</reflink>], p. 245), referred to hereafter as gestures.</p> <p>When exploring gesture use during collaboration, a pertinent factor to consider is whether the gesture accompanies speech or thought. People spontaneously gesture not only during speaking or when describing problem‐solving processes (<emph>co‐speech gesture</emph>) but also while thinking or silently solving problems (Hegarty et al., [<reflink idref="bib17" id="ref20">17</reflink>]; Kita et al., [<reflink idref="bib18" id="ref21">18</reflink>]; Schwartz & Black, [<reflink idref="bib25" id="ref22">25</reflink>]). Chu and Kita ([<reflink idref="bib7" id="ref23">7</reflink>]) classified these types of gestures produced in silent, non‐communicative, problem‐solving situations as <emph>co‐thought gestures</emph>. People produce co‐thought gestures when problem solving is challenging (Chu & Kita, [<reflink idref="bib7" id="ref24">7</reflink>]) or when they reflect on their embodied thoughts (Zurina & Williams, [<reflink idref="bib37" id="ref25">37</reflink>]). For example, Zurina and Williams ([<reflink idref="bib37" id="ref26">37</reflink>]) discovered that students employed co‐thought gestures when encountering dissonance during group work involving fraction problems. Despite being part of a collaborative activity, when students needed to reflect on their own thoughts, they turned their gaze away from the group and made small, silent co‐thought gestures related to the tasks. These gestures were not intended to contribute directly to group discussions, as they were not accompanied by speech. Rather, students used these co‐thought gestures to clarify their own thoughts and aid in comprehending the mathematical task at hand. Zurina and Williams suggested that these "gestures for oneself" were triggered by the need to intensify visuospatial and enactive dimensions of internal thoughts. Given that the distinct role of co‐thought gesture and co‐speech gesture in collaborative settings, it is imperative to consider these gestures as separate entities serving different purposes.</p> <p>Co‐thought gestures remain significantly less comprehended than co‐speech gestures, and their production mechanisms remain largely unknown. Consequently, there is still much to be learned about the specific functions of co‐thought gestures in collaborative learning contexts. Earlier studies have hinted at a shared mechanism underlying the generation of co‐thought and co‐speech gestures, one that activates spatio‐motoric information (Chu & Kita, [<reflink idref="bib7" id="ref27">7</reflink>], [<reflink idref="bib8" id="ref28">8</reflink>]; Kita et al., [<reflink idref="bib18" id="ref29">18</reflink>]). A noteworthy observation is that individuals producing more co‐thought gestures also tend to produce more co‐speech gestures (Chu & Kita, [<reflink idref="bib8" id="ref30">8</reflink>]). Another empirically observed distinctive feature of co‐thought gestures is their smaller, closer‐to‐the‐body movements of hands or arms compared to gestures employed in interpersonal communication (Logan et al., [<reflink idref="bib21" id="ref31">21</reflink>]; Zurina & Williams, [<reflink idref="bib37" id="ref32">37</reflink>]). This observation paves the way for the potential implementation of MMLA techniques in detecting both co‐speech and co‐thought gestures via automatic body movement detection using computer vision tools.</p> <hd id="AN0178994769-5">Automatic detection of body movements and bounding boxes of upper body</hd> <p>Recent advancements in technology have made it possible to automatically capture and detect body movements at high rate (eg, 30 frames per second) and at fine‐grained levels of details (eg, dozens of body landmarks). This technology offers researchers innovative avenues for studying how individuals utilize their bodies during collaboration. In doing so, MMLA researchers collect and analyse skeletal body joint data tracked by Kinect sensors or computer vision tools for the automatic detection of body movements (or gestures) (eg, Echeverría et al., [<reflink idref="bib11" id="ref33">11</reflink>]; Ochoa et al., [<reflink idref="bib23" id="ref34">23</reflink>]; Schneider & Blikstein, [<reflink idref="bib24" id="ref35">24</reflink>]). For instance, Echeverría et al. ([<reflink idref="bib11" id="ref36">11</reflink>]) employed Kinect sensors to capture students' hand and arm movements during presentations and investigated which movement features were indicative of presentation skills. The results revealed that specific features of hand movements, such as the area covered by hands and the smoothness of hand movements, significantly predicted strong presentation skills. The authors then employed a clustering algorithm to automatically classify the detected upper body movements. Similarly, Ochoa et al. ([<reflink idref="bib23" id="ref37">23</reflink>]) collected data on the location of upper body joints (eg, shoulders, elbows, wrists) of presenters to determine whether their body postures were conducive (GOOD) or detrimental (BAD) to effective presentations. Notably, postures marked by body joint positions such as hands in pockets, behind the back, or hands or arms held together (close posture) were classified BAD, while postures putting arms and hands held in an expressive gesture (open posture) were labelled as GOOD. These prior studies suggest that certain forms of body movements and postures can be differentiated based on upper body joint locations.</p> <p>Building upon previous work in the MMLA field, this study endeavours to automatically track learners' upper body movements throughout collaborative multimodal discourse. This entails monitoring changes in body joint positions over time and space. The numeric location data for an individual's upper body joints, including the x‐ and y‐coordinates of the left and right shoulders, elbows, and shoulders, can be used to derive the bounding box of the upper body. This concept of a <emph>bounding box</emph> is commonly used in object detection, in which the objects are wrapped within an imaginary rectangle. A set of coordinates representing the object's smallest enclosing box constitute the bounding box. This concept has been utilized in motion detection studies via computer vision algorithms. For instance, bounding boxes have been employed to detect hand‐raising gestures in real classroom settings (Si et al., [<reflink idref="bib28" id="ref38">28</reflink>]) and to track human pose estimation from videos (Wang et al., [<reflink idref="bib35" id="ref39">35</reflink>]). To our knowledge, no educational research has yet leveraged bounding boxes derived from body joint data to investigate the function of gestures, specifically with regard to the communicative function of co‐speech gestures and the cognitive, non‐communicative function of co‐thought gestures.</p> <p>Empirical observations indicate that co‐thought gestures typically involve smaller hand or arms movement that are closer to the gesturer's body than co‐speech gestures used in interpersonal communication (Logan et al., [<reflink idref="bib21" id="ref40">21</reflink>]; Zurina & Williams, [<reflink idref="bib37" id="ref41">37</reflink>]). Drawing from this insight, the present study posits that a learner exhibiting a <emph>larger variance</emph> in bounding box sizes during the learning process will be more likely to produce more co‐thought and co‐speech gestures, as they generate both small‐ and large‐scale gestures. This hypothesis aligns with the claim that co‐thought and co‐speech gestures emanate from a shared underlying mechanism (Chu & Kita, [<reflink idref="bib7" id="ref42">7</reflink>], [<reflink idref="bib8" id="ref43">8</reflink>]; Kita et al., [<reflink idref="bib18" id="ref44">18</reflink>]). Given that both co‐speech and co‐thought gestures during the learning process represent learners' nonverbal contributions to the co‐construction of embodied math knowledge, a higher occurrence of such gestures implies a deeper level of engagement in the co‐design activity. Conversely, it is posited that learners displaying a <emph>smaller variance</emph> in bounding box sizes during the learning process will be more likely to produce fewer occurrences of both co‐thought and co‐speech gestures, which imply a lower level of engagement in the co‐design activity.</p> <hd id="AN0178994769-6">Intended learning outcomes and epistemic network analysis</hd> <p>A range of empirical studies have consistently shown a positive correlation between learners' engagement and various desirable learning outcomes (eg, Boulton et al., [<reflink idref="bib5" id="ref45">5</reflink>]; Carini et al., [<reflink idref="bib6" id="ref46">6</reflink>]; Lee et al., [<reflink idref="bib20" id="ref47">20</reflink>]). However, existing research on learner engagement and learning outcomes predominantly focuses on individual‐level interactions with learning resources or systems, and their impact on final grades. There remains a gap in the literature concerning how learners' engagement in collaborative peer interactions during learning interventions relates to the desired outcomes of such interventions. In the context of the learning intervention presented in this paper, designed to facilitate PSTs in experiencing the co‐construction of embodied mathematical knowledge, the intended learning outcome is an improvement in PSTs' ability to recognize and interpret multimodal forms of knowledge expression during formative assessment practices. To investigate this, we compare PSTs' multimodal discourse collected from pre‐structured interviews conducted before and after the embodied learning intervention, employing <emph>epistemic network analysis</emph> (ENA). ENA is a discourse analysis technique that identifies and quantifies the connections among cognitive elements in discourse (Shaffer et al., [<reflink idref="bib27" id="ref48">27</reflink>]). Given ENA's utility in revealing distinctions in cognitive structures within discourse through comparative analysis (eg, Sung et al., [<reflink idref="bib29" id="ref49">29</reflink>], [<reflink idref="bib31" id="ref50">31</reflink>]), we expect to gain valuable insights into shifts occurring in PSTs' formative assessment practices resulting from the embodied learning intervention.</p> <hd id="AN0178994769-7">RESEARCH QUESTIONS</hd> <p>In this paper, we investigate how learners' machine‐detected body movements during the learning process relate to their verbal and nonverbal contributions to the co‐construction of embodied math knowledge. The primary objective is to establish the feasibility of utilizing this relatively sparse data as a reasonable indicator for inferring learners' engagement with collaborative knowledge building. Furthermore, we investigate the potential impact of inferred variations in learner engagement on the intended outcomes of the intervention through the application of ENA. Specifically, we focus on changes in how PSTs interpret multimodal forms of knowledge expression during formative assessment practices. To this end, the following research questions guide our investigation:</p> <p>RQ1: How do learners' body movements offer insights into their distinct levels of engagement with the co‐construction of embodied math knowledge during the intervention?</p> <p>This research question gives rise to two specific hypotheses:</p> <hd id="AN0178994769-8">H1a</hd> <p>The learner group displaying larger variances in upper body movements (in comparison to the group with smaller variances) will exhibit statistically significant differences in the number of gestures they produce during the co‐design activity.</p> <hd id="AN0178994769-9">H1b</hd> <p>The learner group displaying larger variances in upper body movements (in contrast to the group with smaller variances) will exhibit statistically significant differences in the number of contextually pertinent verbal utterances they produce during the co‐design activity.</p> <p>RQ2: How do these inferred different levels of engagement potentially impact the intended intervention outcomes?</p> <p>This research question is tied to a particular hypothesis:</p> <hd id="AN0178994769-10">H2</hd> <p>Learner groups displaying larger variances in upper body movements (in comparison to groups with smaller variances) will exhibit significant discourse changes in their interpretations of multimodal forms of knowledge expression during formative assessment practices, before and after the intervention.</p> <p>Through the exploration of these research questions and their corresponding hypotheses, our objective is to illuminate the complex interplay between learners' levels of engagement, machine‐detected and human‐annotated multimodal interactions, and the attainment of desired learning outcomes.</p> <hd id="AN0178994769-11">METHODS</hd> <p>We replicated and modified a pilot study conducted by the authors (Sung et al., [<reflink idref="bib30" id="ref51">30</reflink>]). Drawing on significant and desirable findings from the pilot study, we broadened our investigation to encompass a larger and more diverse sample, specifically from a culturally and linguistically diverse population. Additionally, we extended the analytic scope to incorporate machine‐detected body movement using MMLA techniques. The fundamental components of the initial study design were upheld throughout this process.</p> <hd id="AN0178994769-12">Participants</hd> <p>We recruited K‐12 math PSTs (<emph>N</emph> = 33) from multiple universities in the United States. For the embodied learning intervention, participants were initially divided into groups of four, but the final group size ranged from two to five members due to scheduling challenges. Consequently, we had total of nine groups: one with two members, two with three members, five with four members, and one with five members. Participants received either: (<reflink idref="bib1" id="ref52">1</reflink>) extra course credits (eg, 1 percentage point of course credits) or (<reflink idref="bib2" id="ref53">2</reflink>) monetary compensation (ie, a $100 e‐gift card) upon completion of all study procedure. We provided a certificate of participation in lieu of a monetary compensation for the extra course credit option if the instructors approved of offering extra credit for research participation in their courses and if the participant desired that option. This study was conducted completely online via Zoom, and each participant's individual and collaborative activities were video‐recorded.</p> <hd id="AN0178994769-13">Materials</hd> <p></p> <hd id="AN0178994769-14">The Hidden Village (THV) and Conjecture Editor (THV‐CE)</hd> <p> <emph>The Hidden Village</emph> (THV) is an interactive, 3D motion‐capture simulation program designed to deliver an augmented embodied geometry curriculum through gamified elements. THV aims to promote mathematical reasoning and geometry proof production by imitating the in‐game avatar's cognitively relevant body movements called <emph>directed actions</emph>. Within THV, the THV Conjecture Editor (THV‐CE) serves as a built‐in program, acting as a design tool for creating new directed actions. In this study, THV‐CE played a central role in the co‐design activity, where participants collaboratively generated mathematically relevant directed actions using the editable avatar's upper body movements (arms and hands), with the objective of fostering embodied mathematical reasoning (see Figure 1). Given the virtual setting of the co‐design activity, group members expressed their ideas verbally and gesturally within the Zoom platform, while a researcher acted as a proxy to operate THV‐CE under participants' directions.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep24/bjet13473-fig-0001.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13473-fig-0001.jpg" title="1 Participants discussing to create new directed actions during the co‐design activity." /> </p> <p></p> <hd id="AN0178994769-16">Pre‐ and post‐interviews</hd> <p>Before and after the intervention, participants viewed one‐minute videos featuring a student explaining the reasons behind a certain geometric conjecture being either sometimes false or always true. To uphold privacy, the original student videos were re‐enacted by an actor, as depicted in Figure 2. Following the viewing of these actor‐based videos, participants responded to semi‐structured interview questions designed to elicit their interpretations and formative assessments of the student's understanding of mathematical concepts. During the interviews, participants were prompted to provide explicit evidence derived from the video content. Therefore, these pre‐ and post‐interviews were utilized to assess whether there were any changes in PSTs' formative assessment practices after experiencing the embodied learning intervention.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep24/bjet13473-fig-0002.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13473-fig-0002.jpg" title="2 A screenshot of a student video (re‐enacted by an actor) used during semi‐structured pre‐ and post‐interviews." /> </p> <p></p> <hd id="AN0178994769-18">Procedures</hd> <p>Throughout the intervention, participants engaged in a 3.5‐hour‐long online session that contained a series of activities, including (<reflink idref="bib1" id="ref54">1</reflink>) a pre‐intervention video‐prompted individual interview, (<reflink idref="bib2" id="ref55">2</reflink>) experiencing online gameplay of THV, (<reflink idref="bib3" id="ref56">3</reflink>) engaging in co‐design activity conducted in groups using THV‐CE, and (<reflink idref="bib4" id="ref57">4</reflink>) a post‐intervention video‐prompted interview. While all participants completed the entire set of activities, our study specifically focuses on analysing multimodal discourse data from the co‐design activity and pre‐ and post‐interviews.</p> <p>This study strictly adhered to ethical guidelines, with participants providing written informed consent before engaging in any research activities. The Institutional Review Board (IRB) approved the research protocol, ensuring participant confidentiality. Withdrawal rights were communicated, and data were securely stored, accessible only to the research team. More information on ethical considerations can be found in the <emph>Ethical Statement</emph> section.</p> <hd id="AN0178994769-19">Data analysis</hd> <p></p> <hd id="AN0178994769-20">Upper body movement variance using machine‐detected body joint data</hd> <p>Throughout the co‐design activity, we tracked learners' upper body movements, encompassing both gestural and non‐gestural motions. <emph>PoseNet</emph>, a web‐based motion‐sensing tool equipped with advanced computer vision systems (Hassan et al., [<reflink idref="bib16" id="ref58">16</reflink>]), was employed to capture these data. This technology provided the automated tracking of alterations in body joint positions at a high rate of 30 frames per second. PoseNet was chosen over alternatives like MediaPipe due to its superior in real‐time human pose estimation accuracy (Guo et al., [<reflink idref="bib15" id="ref59">15</reflink>]), aligning well with our objective of automatically detecting body movement during the co‐design activity. Notably, we analysed individual video data of each participant captured by their laptop webcam, thereby all video recordings featured a single person in each instance.</p> <p>In order to estimate the dimension of a bounding box enclosing the upper body, we calculated its width and height by measuring the distance between the maximum and minimum x‐ and y‐coordinate values of an individual's 6 body joint positions (eg, left and right shoulders, elbows, and wrists), and then multiplied these values (see Figure 3). The variance of each learner's upper body movements was measured separately for each of the three given conjectures during the co‐design activity, resulting in a total of 99 observations (variances) corresponding to the 33 participants.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep24/bjet13473-fig-0003.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13473-fig-0003.jpg" title="3 Estimation of the bounding box size enclosing the upper body using the body joint positions (x‐ and y‐coordinates of the left and right shoulders, elbows, and wrists)." /> </p> <p></p> <p>In this study, the variance of a learner's upper body movements was characterized by the variance of their bounding box dimensions. To elucidate the relationship between learners' upper body movements and verbal and nonverbal interactions that occurred during the co‐design activity, we divided learners into two groups based on the median value of variances in bounding box dimensions: (a) those with larger variances in upper body movements (henceforth, larger variances group) and (b) those with smaller variances (henceforth, smaller variances group).</p> <hd id="AN0178994769-22">Multimodal discourse coding: Co‐design activity</hd> <p>In order to investigate how learners' body movements offer insights into their distinct levels of engagement with the co‐construction of embodied math knowledge during the intervention (RQ1), we transcribed learners' speech and gestures from the co‐design activity. The initial transcription of learners' speech was generated by the automatic audio transcription tool supplied by Zoom. Subsequently, we annotated learners' gestures by closely observing the video recordings in conjunction with instances of speech. This procedure yielded multimodal transcripts of the co‐design activity that were delineated into segments corresponding to events of verbal, gestural, or multimodal contribution. An event was defined as a learner's expression that occurred without a pause, a total of 3623 segments. These segments encapsulated scenarios where learners generated either a verbal utterance alone, a gesture alone (co‐thought gesture), or a fusion of both verbal utterance and co‐speech gesture without any pauses.</p> <p>The transcripts of learners' verbal speech and gestures were subjected to distinct coding methodologies for each modality. First, we coded the occurrences of verbal utterances that were contextually pertinent to the co‐design activity. These verbal codes were derived from the speech transcripts using the following three categories: encompassing discussions involving (<reflink idref="bib1" id="ref60">1</reflink>) mathematical concepts like angles, lines, and conjectures (mathematical thinking); (<reflink idref="bib2" id="ref61">2</reflink>) visualizing geometric conjectures and designing directed actions to help students' mathematical reasoning (design oriented); and (<reflink idref="bib3" id="ref62">3</reflink>) collaborative efforts aimed at designing directed actions for given conjectures (consensus building). Utilizing an automated process based on regular expression matching techniques (nCoder; Marquart et al., [<reflink idref="bib22" id="ref63">22</reflink>]), the inter‐rater reliability for the three verbal codes was established through pairwise comparisons between two human raters and nCoder. The Cohen's kappa scores surpassed the threshold (kappa > 0.80).</p> <p>For gesture coding, we first determined whether a gesture occurred, and if so, whether it fits either of the well‐established gesture types of iconic (shape‐resembling) or metaphoric (concept‐resembling) gestures that together make up the representational gesture category (Alibali & Nathan, [<reflink idref="bib2" id="ref64">2</reflink>]). We then distinguished between gestures that accompanied speech (co‐speech gestures) and those that occurred during moments of thought without speech (co‐thought gestures). The Cohen's kappa scores between two human raters for these gesture codes exceeded the threshold (kappa >0.80).</p> <hd id="AN0178994769-23">Multimodal discourse coding: Pre‐ and post‐interviews</hd> <p>To examine how the inferred different levels of engagement with the intervention impact the intended intervention outcomes (RQ2), we transcribed learners' speech and gestures during the pre‐ and post‐interviews. Similar to the co‐design activity, the multimodal transcripts of pre‐ and post‐interviews were segmented at the level of utterances. An utterance was defined as a learner's statement made without any pauses, leading to a total of 3855 segments. Given that all gestures produced during individual interviews accompanied speech, each line in these multimodal transcripts encapsulated an instance where a learner generated a verbal utterance or a combination of a verbal utterance and a co‐speech gesture, without pauses.</p> <p>For the process of discourse coding, we adopted a grounded theory approach (Glaser & Strauss, [<reflink idref="bib14" id="ref65">14</reflink>]) to uncover key epistemic concepts that emerged from the data at the segment level. This bottom‐up, grounded process identified six discourse codes applied to each segment of the multimodal transcripts (see Table 1). The inter‐rater reliability for multimodal and verbal codes was established independently by comparing pairwise agreement between two human coders and nCoder, whereas for gestural codes it was established between the two human coders. Pairwise Cohen's kappa scores ranged between 0.83 ≤ <emph>κ</emph> ≤ 0.97 for each code, with all kappa values having Shaffer's rho values <emph>ρ</emph> < 0.05 (Shaffer, [<reflink idref="bib26" id="ref66">26</reflink>]). It means that if the coders were to code the whole dataset, they would have a level of agreement of kappa > 0.80 with a Type I error rate of less than 5%. While transcript segments could receive multiple codes, the two gestural codes were mutually exclusive.</p> <p>1 TABLE Coding scheme for multimodal discourse during the pre‐ and post‐interviews, including code descriptions, examples, and inter‐rater reliability statistics.</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Code</th><th align="left">Description</th><th align="left">Example</th></tr></thead><tbody valign="top"><tr><td align="left"><sc>Multimodal: mathematical thinking</sc></td><td align="left">PSTs' multimodal expressions of mathematical concepts such as angles, lines, and conjectures via either speech, gesture, or both</td><td align="left">"Start with a larger one [making a V shape using two arms] showing the larger angle results in a larger side length and then showing that a small angle [making a smaller angle by putting arms closer] will result in a smaller side length"</td></tr><tr><td align="left"><sc>Multimodal: pedagogical use of gesture</sc></td><td align="left">PSTs' multimodal expression of how they use (or will use) gestures pedagogically via either speech, gesture, or both. This includes interpreting the meaning of students' gestures, focusing on relationship between gestures and speech in student reasoning, or planning on gesture use in future instruction</td><td align="left">"Being able to have them to show them the proper gestures that way they could get their ideas out a lot easier [making a diagonal line by slanting the right arm]"</td></tr><tr><td align="left"><sc>Verbal: assessment</sc></td><td align="left">PSTs' judgement on the level of students' understanding</td><td align="left">"I don't think she fully comprehended the concept behind why they would be equivalent"</td></tr><tr><td align="left"><sc>Verbal: verbal evidence</sc></td><td align="left">PSTs' use of students' utterances as reference or evidence</td><td align="left">"When she's explaining why it wouldn't work, she's basically just re‐reading the conjecture too"</td></tr><tr><td align="left"><sc>Gestural: gesture replay</sc></td><td align="left">PSTs' actions of replaying observed students' gestures</td><td align="left">"It seemed like she understood the underlying principle of the conjecture statement, which was the opposite angles, if the lines are crossed [crossing both arms] are always the same"</td></tr><tr><td align="left"><sc>Gestural: spontaneous gesture</sc></td><td align="left">PSTs' actions of producing spontaneous gestures</td><td align="left">"A bigger angle creates a bigger opening [using both hands to make an angle and showing the length of the side opposite the angle]"</td></tr></tbody></table> </ephtml> </p> <p>1 <emph>Note</emph>: Quotations are verbal utterances, and square brackets [...] indicate gestures.</p> <hd id="AN0178994769-24">ENA models for the discourse comparison between the pre‐ and post‐interviews</hd> <p>The coded multimodal transcripts of pre‐ and post‐interviews were analysed using ENA. ENA builds dynamic discourse models: one takes the form of a nodal network, wherein the network's edges signify the relative frequency of co‐occurrence between two codes; the other involves a plotted points graph that visualizes the position of individual networks within the projected ENA space. This positioning is determined by calculating a mean centroid around which discourse centres, with connections between codes being weighted (Shaffer, [<reflink idref="bib26" id="ref67">26</reflink>]).</p> <p>To ascertain potential differences in discourse patterns between pre‐ and post‐interviews among the two learner groups (larger or smaller variances groups), we conducted a two‐tailed paired <emph>t</emph>‐test to compare the positions of plotted points within the projected ENA space. We also created the corresponding network graphs to visually elucidate which connections among the codes account for any significant differences. In essence, we compared ENA discourse models between the following pairings: (<reflink idref="bib1" id="ref68">1</reflink>) pre‐ and post‐interviews of the larger variances group, and (<reflink idref="bib2" id="ref69">2</reflink>) pre‐ and post‐interviews of the smaller variances group.</p> <hd id="AN0178994769-25">RESULTS</hd> <p></p> <hd id="AN0178994769-26">RQ1: How do learners' body movements offer insights into their distinct levels of engagement...</hd> <p>To address this question, we first clarified the interpretation of learn groups with larger and smaller variances. Given that the variance of a learner's upper body movements was defined by the variance of their bounding box dimensions, this implies that larger variances in the bounding box sizes represent a more diverse range of upper body movements. Such movements could include both small movements often associated with co‐thought gestures and much larger communicative movements involving the whole upper body. Conversely, smaller variances in the bounding box sizes suggest movement confined to a particular region. However, this presents two possible contradictory interpretations: learners either predominantly engage in large movements linked with speech or maintain a seated posture marked by limited upper body movements. Thus, we conducted a Pearson's correlation test to examine the relationship between the variance of a learner's bounding box sizes and the average bounding box sizes exhibited during the learning process. The results revealed a strong positive correlation between the two (<emph>r</emph>(<reflink idref="bib97" id="ref70">97</reflink>) = 0.53, <emph>p</emph> < 0.01), indicating that smaller variances tended towards one region of movement, potentially reflecting a more stationary posture with diminished upper body motions (see Figure 4).</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep24/bjet13473-fig-0004.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13473-fig-0004.jpg" title="4 Identifying learner groups with larger and smaller variances in upper body movements." /> </p> <p></p> <p>Expanding on the identified characteristics of each learner group, we conducted a comparison of the number of verbal and nonverbal interactions generated by groups with larger variances versus smaller variances during the co‐design activity. The number of learner's verbal and nonverbal interactions was quantified based on the occurrences of gestural and verbal codes. In light of the anticipated directional differences (Coolican, [<reflink idref="bib9" id="ref71">9</reflink>])—specifically, expecting that larger variances in upper body movements would lead to a higher number of co‐speech and co‐thought gestures due to the inclusion of both small‐ and large‐scale gestures—we computed a one‐tailed independent <emph>t</emph>‐test. Table 2 provides the resulting statistics between the two learner groups.</p> <p>2 TABLE T‐test results of the number of gestures and verbal utterances between the learner groups with larger (versus smaller) variances in upper body movements (N  = 99).</p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left" /><th align="left">Larger variances group</th><th align="left">Smaller variances group</th><th align="left"><italic>t</italic></th><th align="left"><italic>p</italic>‐value</th><th align="left">Cohen's <italic>d</italic></th></tr><tr><th align="left"><italic>M</italic></th><th align="left"><italic>SD</italic></th><th align="left"><italic>M</italic></th><th align="left"><italic>SD</italic></th></tr></thead><tbody valign="top"><tr><td align="left">Total number of gestures</td><td align="char" char=".">11.4</td><td align="char" char=".">9.09</td><td align="char" char=".">8.18</td><td align="char" char=".">6.72</td><td align="char" char=".">2.01</td><td align="char" char=".">0.02<xref ref-type="fn" rid="tfn2" /></td><td align="char" char=".">0.40</td></tr><tr><td align="left">Co‐thought gestures</td><td align="char" char=".">3.68</td><td align="char" char=".">4.70</td><td align="char" char=".">2.37</td><td align="char" char=".">2.64</td><td align="char" char=".">1.72</td><td align="char" char=".">0.05<xref ref-type="fn" rid="tfn2" /></td><td align="char" char=".">0.34</td></tr><tr><td align="left">Co‐speech gestures</td><td align="char" char=".">7.72</td><td align="char" char=".">7.43</td><td align="char" char=".">5.82</td><td align="char" char=".">5.57</td><td align="char" char=".">1.44</td><td align="char" char=".">0.08</td><td align="char" char=".">0.29</td></tr><tr><td align="left">Total number of verbal utterances</td><td align="char" char=".">7.44</td><td align="char" char=".">6.53</td><td align="char" char=".">5.35</td><td align="char" char=".">3.86</td><td align="char" char=".">1.95</td><td align="char" char=".">0.03<xref ref-type="fn" rid="tfn2" /></td><td align="char" char=".">0.40</td></tr><tr><td align="left">Mathematical thinking</td><td align="char" char=".">5.08</td><td align="char" char=".">5.24</td><td align="char" char=".">3.53</td><td align="char" char=".">3.34</td><td align="char" char=".">1.76</td><td align="char" char=".">0.04<xref ref-type="fn" rid="tfn2" /></td><td align="char" char=".">0.35</td></tr><tr><td align="left">Design oriented</td><td align="char" char=".">1.94</td><td align="char" char=".">3.16</td><td align="char" char=".">1.39</td><td align="char" char=".">1.40</td><td align="char" char=".">1.13</td><td align="char" char=".">0.13</td><td align="char" char=".">0.23</td></tr><tr><td align="left">Consensus building</td><td align="char" char=".">2.28</td><td align="char" char=".">1.95</td><td align="char" char=".">1.71</td><td align="char" char=".">1.37</td><td align="char" char=".">1.67</td><td align="char" char=".">0.05<xref ref-type="fn" rid="tfn2" /></td><td align="char" char=".">0.34</td></tr></tbody></table> </ephtml> </p> <p>2 * <emph>p</emph> < 0.05.</p> <p>The results unveiled a statistically significant distinction in the total number of gesture production between the larger variances group and smaller variances group (H1a; <emph>t</emph> = 2.01, <emph>p</emph> < 0.05). During the co‐design activity, the larger variances group produced significantly more gestures (<emph>M</emph> = 11.4, <emph>SD</emph> = 9.09) than the smaller variances group (<emph>M</emph> = 8.18, <emph>SD</emph> = 6.72), with a medium effect size (Cohen's <emph>d</emph> = 0.40). Specifically, among the types of gestures, the larger variances group exhibited significantly more co‐thought gestures (<emph>M</emph> = 3.68, <emph>SD</emph> = 4.70) than the smaller variances group (<emph>M</emph> = 2.37, <emph>SD</emph> = 2.64), accompanied by a medium effect size (Cohen's <emph>d</emph> = 0.34, <emph>t</emph> = 1.72, <emph>p</emph> < 0.05).</p> <p>Moreover, the results indicated a statistically significant difference in the total number of verbal utterances produced by each learner group (H1b; <emph>t</emph> = 1.95, <emph>p</emph> < 0.05). During the co‐design activity, the larger variances group generated significantly more verbal utterances (<emph>M</emph> = 7.44, <emph>SD</emph> = 6.53) compared to the smaller variances group (<emph>M</emph> = 5.35, <emph>SD</emph> = 3.86), resulting in a medium effect size (Cohen's <emph>d</emph> = 0.40). Focusing on specific categories of verbal utterances, the numbers pertaining to mathematical concepts (<emph>t</emph> = 1.76, <emph>p</emph> < 0.05) and consensus building (<emph>t</emph> = 1.67, <emph>p</emph> < 0.05) exhibited significant differences between the two learner groups. The larger variances group produced significantly more verbal utterances regarding mathematical concepts (<emph>M</emph> = 5.08, <emph>SD</emph> = 5.24) and consensus building (<emph>M</emph> = 2.28, <emph>SD</emph> = 1.95) than the smaller variances group (<emph>M</emph> = 3.34, <emph>SD</emph> = 1.76; <emph>M</emph> = 1.37, <emph>SD</emph> = 1.67, respectively). Both effect sizes were moderate (Cohen's <emph>d</emph> = 0.35 and 0.34, respectively).</p> <hd id="AN0178994769-28">RQ2: How do these inferred different levels of engagement potentially impact the intended int...</hd> <p>To explore how these inferred different levels of engagement potentially impact the intended intervention outcomes, we investigated any discourse changes while PSTs in each learner group (larger versus smaller variances groups) formatively assessing students' multimodal forms of math knowledge during the pre‐ and post‐interviews. Using mixed methods, we first conducted a quantitative analysis using ENA to identify any distinctive discourse shifts between the pre‐ and post‐interviews in each learner group. Subsequently, we supplemented the quantitative findings with qualitative analyses by offering illustrative instances.</p> <hd id="AN0178994769-29">Quantitative results using ENA models</hd> <p>We created the ENA scatter plot (ie, a graph of plotted points) to examine whether statistically significant changes existed between pre‐ and post‐interviews of PSTs in the larger variances group. In Figure 5, each circular plotted point represents an individual PST's network location in pre‐interviews (in red) and post‐interviews (blue), based on the weighted average of node weights within each network. This allowed us to gauge group means represented by larger square points (1 red, 1 blue), accompanied by 95% confidence intervals (<emph>t</emph>‐distribution) visualized through dashed boxes in red and blue, respectively. Non‐overlapping intervals would suggest statistically significant differences in means at the 5% level. The statistical analysis confirmed significant divergence in discourse patterns between pre‐ and post‐interviews of PSTs in the larger variances group, displaying a considerable effect size ( <ephtml> <math altimg="urn:x-wiley:00071013:media:bjet13473:bjet13473-math-0001" display="inline" overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>X</mi><mo>¯</mo></mover></mrow></semantics></math> </ephtml><subs>Larger_Pre</subs> = −0.66, <ephtml> <math altimg="urn:x-wiley:00071013:media:bjet13473:bjet13473-math-0002" display="inline" overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>X</mi><mo>¯</mo></mover></mrow></semantics></math> </ephtml><subs>Larger_post</subs> = 0.66, <emph>t</emph>(18.97) = −3.19, <emph>p</emph> < 0.01, Cohen's <emph>d</emph> = 1.36).</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep24/bjet13473-fig-0005.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13473-fig-0005.jpg" title="5 ENA scatterplot of PSTs in the larger variances group, portraying individual PST's network locations of pre‐interviews (red) and post‐interviews (blue)." /> </p> <p></p> <p>To gain insights and visual clarity into the changes in connections among codes between the pre‐ and post‐interviews, we conducted a comparative analysis of mean epistemic networks between pre‐ and post‐interviews of PSTs in the larger variances group, as depicted in Figure 6. The mean subtracted network (Panel b of Figure 6) distinctly captures the <emph>changes</emph> in PSTs' discourse pertaining to formative assessment practices consequent to the intervention. This visualization method accentuates the distinctions in connection weights between the pre‐interview mean epistemic network (Panel a), reflecting their initial interpretations, and the post‐interview mean epistemic network (Panel c). Panel b of Figure 6 shows that during the pre‐interviews (marked by red connections), PSTs in the larger variances group exhibited moderately strong connections between gesture replay, mathematical thinking, and assessment. This indicates that PSTs in the larger variances group were initially preoccupied with replaying observed students' gestures (gesture replay) to derive simple inferences (assessment) between representing gestures and possessing mathematical knowledge (mathematical thinking). On the contrary, during the post‐intervention interviews (marked by blue connections in Panel b of Figure 6), PSTs in the larger variances group displayed strong connections between verbal evidence, pedagogical use of gesture, and mathematical thinking. This suggests that following the intervention, PSTs in the larger variances group became more aware of the importance of integrating information from students' nonverbal cues with speech (verbal evidence), focusing on interpreting students' multimodal knowledge expressions (pedagogical use of gesture) while formatively assessing their mathematical understanding (mathematical thinking).</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep24/bjet13473-fig-0006.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13473-fig-0006.jpg" title="6 Mean epistemic network graphs of PSTs in the larger variances group, showing the connections made in pre‐interviews (red lines of Panel a), post‐interviews (blue lines of Panel c), and mean subtracted network (Panel b)." /> </p> <p></p> <p>On the other hand, when we created the ENA scatter plot of PSTs in the smaller variances group (see Figure 7), the statistical analysis indicated that there were no significant changes in discourse patterns between pre‐ and post‐interviews ( <ephtml> <math altimg="urn:x-wiley:00071013:media:bjet13473:bjet13473-math-0003" display="inline" overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>X</mi><mo>¯</mo></mover></mrow></semantics></math> </ephtml><subs>Smaller_Pre</subs> = −0.18, <ephtml> <math altimg="urn:x-wiley:00071013:media:bjet13473:bjet13473-math-0004" display="inline" overflow="scroll" xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mover accent="true"><mi>X</mi><mo>¯</mo></mover></mrow></semantics></math> </ephtml><subs>Smaller_post</subs> = 0.18, <emph>t</emph>(41.65) = 1.03, <emph>p</emph> > 0.05, Cohen's <emph>d</emph> = 0.31). Consequently, we refrained from the construction of mean epistemic networks between the pre‐ and post‐interviews of PSTs in the smaller variances group.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep24/bjet13473-fig-0007.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13473-fig-0007.jpg" title="7 ENA scatterplot of learners with smaller variances in upper body movements, showing plotted points of individual PST's networks of pre‐interviews (red) and post‐interviews (blue)." /> </p> <p></p> <hd id="AN0178994769-33">Qualitative results</hd> <p>In order to attain a more nuanced and holistic understanding of the distinctive differences in the desired intervention outcomes, we qualitatively analysed the pre‐ and post‐interviews of PSTs in the larger variances group, using the coded multimodal transcript of the interviews.</p> <p>Figure 8 illustrates an excerpt and corresponding screenshots from the pre‐interview with PST‐12. In this instance, PST‐12 replays the student's gestures shown in the video (Panels a and b in Figure 8; gesture replay code) and describes the geometric objects represented by such gestures ("two lines that cross" and "the opposite angles"; mathematical thinking). This PST‐12's description demonstrates an inherent ability to discern and interpret visual information conveyed through gestures while sharing mathematical ideas. However, in line 2, PST‐12 draws a preliminary conclusion that the student "understood for the most part" (assessment) of the mathematical concept (in this case, opposite rule), seemingly without exploring how the student's gestures contribute to their reasoning process. Evidently, PST‐12 makes a simple inference that the students' gestures, depicting relevant geometric objects for a given conjecture, tie to a profound grasp of the mathematical concept. This instance exemplifies how PSTs in the larger variances group interpret the student's math knowledge during the pre‐interviews—they tend to regard student gestures as direct evidence of mathematical understanding. In this interpretation, PSTs' focus seems to be predominantly on replaying the gestures themselves, with relatively limited attention paid to the interplay between verbal and nonverbal modes of knowledge expression during the reasoning process.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep24/bjet13473-fig-0008.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13473-fig-0008.jpg" title="8 Excerpt of PST‐12's response in post‐interviews: Gesture replays of (a) crossing arms gesture and (b) pointing at opposite angles." /> </p> <p></p> <p></p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left">Line</th><th align="left">Time</th><th align="left">Multimodal transcript</th><th align="left">Codes</th></tr></thead><tbody valign="top"><tr><td align="left">1</td><td align="left">01:38–01:47</td><td align="left">From what she did with her hands [replaying the student's crossing arms gesture], she was showing two lines that cross, and then showed that the opposite angles [pointing left and right sides of the opposite angles using both hands] were the same</td><td align="left"><sc>gesture replay, pedagogical use of gesture, mathematical thinking</sc></td></tr><tr><td align="left">2</td><td align="left">01:48–01:56</td><td align="left">So that led me to believe that she understood for the most part of what it was, or what the statement meant</td><td align="left"><sc>assessment, mathematical thinking</sc></td></tr></tbody></table> </ephtml> </p> <p>Subsequent to the embodied collaborative intervention, PSTs were exposed to another actor video in which the actor portrayed a different student reasoning about the same geometric conjecture. Figure 9 encapsulates an excerpt from the post‐interview with PST‐21, serving as an illustrative exemplar of the changes in PSTs' formative assessment practices after the intervention.</p> <p> <img src="https://imageserver.ebscohost.com/img/embimages/rdk/58I/01sep24/bjet13473-fig-0009.jpg?ephost1=dGJyMNXb4kSepq84yOvqOLCmsE6epq5Srqa4SK6WxWXS" alt="bjet13473-fig-0009.jpg" title="9 Excerpt of PST‐21's response in post‐interviews: Gesture replays of (a) crossing hands and (b) hand bent gestures." /> </p> <p></p> <p>In line 1 of Figure 9, similar to the aforementioned instance, PST‐21 begins by replicating the student's gestures shown in the video (gesture replay). However, PST‐21's approach to evaluating the student's understanding of the mathematical concept is distinctive from PST‐12 in the pre‐interviews. While reproducing and interpreting the student's gesture (Panels a and b; gesture replay and pedagogical use of gesture in line 2), PST‐21 simultaneously articulates the student's verbal contribution to the reasoning, saying "she mentioned that if one was bigger, then it would have to curve the line" (verbal evidence and mathematical thinking). This underscores PST‐21 adeptness in identifying the core logical underpinning of the student's reasoning by incorporating information derived from both their verbal and nonverbal ways of knowledge expressions. Overall, during the post‐intervention interviews, PSTs in the larger variances group evidently exhibit an enhanced proficiency in interpreting multimodal forms of math knowledge while formatively assessing students' understanding of the math concepts.</p> <p>These qualitative findings corroborated the quantitative findings from ENA models.</p> <p></p> <p> <ephtml> <table><thead valign="bottom"><tr><th align="left" /><th align="left">Time</th><th align="left">Multimodal transcript</th><th align="left">Codes</th></tr></thead><tbody valign="top"><tr><td align="left">1</td><td align="left">00:18–00:26</td><td align="left">She was doing diagonals with her hands [mimicking the student's crossing hands gesture], which showed that she was thinking about the intersecting lines</td><td align="left"><sc>gesture replay, pedagogical use of gesture, mathematical thinking</sc></td></tr><tr><td align="left">2</td><td align="left">00:26–00:40</td><td align="left">And how the opposite angles [pointing upper and bottom sides of the opposite angles using the right hand while holding the left arm diagonally] would be the same, and she mentioned that if one was bigger, then it would have to curve the line [mimicking the student's hand bent gesture], which I thought was interesting, and she was using a lot of gesture, so I think it did show that she did understand the concepts</td><td align="left"><sc>gesture replay, pedagogical use of gesture, assessment, verbal evidence, mathematical thinking</sc></td></tr></tbody></table> </ephtml> </p> <hd id="AN0178994769-36">DISCUSSION</hd> <p>This study investigated the relationships between learners' machine‐detected body movements during the learning process and their verbal and nonverbal contributions to the co‐construction of embodied math knowledge. Our principal aim is to substantiate the feasibility of employing this relatively sparse data as a valid indicator for inferring learners' engagement with the collaborative knowledge creation process. In addition, the exploration extended to investigating the potential impacts of the inferred distinct levels of learner engagement on the intended outcomes of the intervention, namely the changes in PSTs' abilities to interpret multimodal forms of knowledge expression during formative assessment practices.</p> <p>Our findings relating to RQ1 demonstrated that the variance in upper body movements, which can be automatically tracked in real time by machine, has the potential to serve as a reasonable indicator for learners' multimodal engagement with collaborative knowledge‐building endeavours. Notably, the learner group characterized by larger variances in upper body movements produced significantly more gestures and contextually relevant verbal utterances during the co‐design activity than the group displaying smaller variances. These findings provide empirical support for the hypotheses derived from prior literature (Chu & Kita, [<reflink idref="bib7" id="ref72">7</reflink>], [<reflink idref="bib8" id="ref73">8</reflink>]; Kita et al., [<reflink idref="bib18" id="ref74">18</reflink>]), affirming that learners with larger variances in upper body movements are likely to manifest more occurrences of both gestures (H1a) and verbal utterances (H1b). This observed pattern of learning behaviours serves as a potent indicator of deeper engagement with collaborative creation of embodied knowledge during the learning process. Importantly, our investigation goes beyond the scope of examining the relationship solely between machine‐detected body movement data and human‐annotated gesture data within the same modality. We investigated the interplay of these body movement data with verbal contributions to the co‐construction of embodied knowledge, aiming to uncover <emph>cross‐modal</emph> connections—central interests of the MMLA field.</p> <p>Our findings relating to RQ2 empirically validated that learners' distinct levels of engagement with the intervention, inferred by the variances in upper body movements, directly affected the desired intervention outcomes. Leveraging ENA models coupled with qualitative analysis, we unveiled that PSTs in the learner group with larger variances group demonstrated significant changes in how they interpreted multimodal forms of math knowledge during formative assessment practices between before and after the intervention. In contrast, PSTs in the learner group with smaller variances, indicating comparatively lower engagement, showed no significant changes in the discourse patterns of formative assessment practices between pre‐ and post‐interviews. These results are consistent with prior research indicating a positive correlation between learners' engagement and various desirable learning outcomes (eg, Boulton et al., [<reflink idref="bib5" id="ref75">5</reflink>]; Carini et al., [<reflink idref="bib6" id="ref76">6</reflink>]; Lee et al., [<reflink idref="bib20" id="ref77">20</reflink>]). They further reinforce the established premise of this study, underscoring that relatively thin data about variances in machine‐detected body movements can effectively indicate learners' engagement levels with the collaborative embodied intervention, significantly shaping the desired learning outcomes. By providing empirical evidence on how learners' engagement in collaborative peer interactions during the learning intervention relates to individual‐level desired learning outcomes, this fills a gap in the existing literature. It emphasizes the significance of individual engagement with the collaborative learning process during learning interventions for achieving intended learning outcomes, a dimension that has been relatively underexplored in previous studies.</p> <hd id="AN0178994769-37">LIMITATIONS AND FUTURE STUDIES</hd> <p>This study has some limitations. One is that we examined learners' multimodal discourse within a specific domain of mathematics. Future research is needed to explore how this finding is transferable to other mathematical domains and other academic content areas. Second, the present coding system utilized for this investigation may have limitations. For instance, pedagogical use of gesture, the gestural code from pre‐ and post‐interviews, may capture too many things simultaneously, as it includes PSTs' interpretation of the meaning of students' gestures and their plan on gesture use in future instruction. This code needs to be revised in the future work to elucidate the different pedagogical uses of gesture by teachers in various contexts. Third, given that multiple factors of participants, such as sociocultural backgrounds, gender, and personality, can contribute to differences in one's use of gesture space, which was measured by the dimensions of the bounding box enclosing one's upper body, variances in bounding box areas may vary in other samples from different populations. Fourth, the online learning environment in our study could influence the nature of gestures, engagement patterns, and overall collaborative interactions compared to face‐to‐face environments where learners physically co‐located. Future research should delve deeper into understanding the nuanced effects of the online environment on collaborative learning dynamics, considering potential differences in gestures and interactions that may emerge in virtual spaces compared to physical classrooms. Lastly, the relatively low sample size (<emph>N</emph> = 33) raises concerns about generalizability, a challenge commonly encountered in ENA studies. Elmoazen et al. ([<reflink idref="bib13" id="ref78">13</reflink>]) highlighted the prevalence of such sample sizes in ENA research, noting that most published studies include fewer than 100 students, with a median of 32 students (as found in 82% of studies in the systematic review of ENA studies). While the existence of studies with similar sample sizes does not inherently address generalizability concerns, it does, to some extent, mitigate them, given the field's widespread acceptance of and operation within such sample size constraints. Finally, the distribution of students per group might affect the final results, potentially impacting the overall dynamics of collaborative interactions. In future research, it is advisable to explore additional dimensions in analysis, such as the quality of group interactions, alongside expanding the participant pool to encompass larger and more diverse groups. This broader approach will enhance the generalizability of findings, taking into account potential impact of group dynamics on the outcomes.</p> <hd id="AN0178994769-38">CONCLUSION</hd> <p>This study fosters a more comprehensive understanding of the interplay between verbal and nonverbal elements in collaborative knowledge construction. By focusing on the role of gestures in collaborative learning contexts, we address an identified research gap in the literature and provide empirical evidence of their significance in facilitating both <emph>intrapersonal</emph> and <emph>interpersonal</emph> cognitive processes during group work. This insight is particularly crucial for promoting more equitable approaches in assessment and collaboration, mitigating the potential risk of undervaluing individuals' nonverbal engagement and contributions, especially among students from linguistically diverse backgrounds (Lee & Fradd, [<reflink idref="bib19" id="ref79">19</reflink>]). Furthermore, our research showcases the feasibility of utilizing machine‐detected body movements as a viable indicator of engagement in collaborative learning, leveraging MMLA techniques that streamline gesture analysis. This innovative methodology not only paves the way for investigating collaborative learning dynamics but also enables the assessment of individual engagement during the learning process, thereby enhancing support for learners to achieve desired learning outcomes. Hence, our study makes valuable contributions to our scientific understanding of multimodal ways of knowledge expression and assessment in learning, teaching, and collaboration. These insights carry implications for practitioners, including researchers and teachers, who seek to improve collaborative learning experiences and ensure accurate and equitable assessment of student engagement in collaboration across diverse learning environments.</p> <hd id="AN0178994769-39">FUNDING INFORMATION</hd> <p>This work received no external funding. All aspects of the study, including design, data collection, analysis, interpretation, and manuscript preparation, were carried out without the support of external financial assistance.</p> <hd id="AN0178994769-40">CONFLICT OF INTEREST STATEMENT</hd> <p>The authors declare no conflicts of interest that could have influenced the design, execution, or reporting of this research.</p> <hd id="AN0178994769-41">DATA AVAILABILITY STATEMENT</hd> <p>The data that support the findings of this study are available upon reasonable request from the corresponding author. However, due to the sensitive nature of some of the data and the need to protect participant confidentiality, certain restrictions may apply to the availability of specific data.</p> <hd id="AN0178994769-42">ETHICS STATEMENT</hd> <p>Every participant involved in this study provided written informed consent before their participation, with a focus on maintaining the confidentiality of their identities throughout the research process. This consent encompassed individuals' approval for the potential publication of any identifiable images or data included in this article. The research protocol was approved by the IRB, ensuring adherence to ethical principles. Participants were informed of their rights to withdraw from the study at any point, and the approved protocols were in place to securely store and restrict access to the collected data, which was solely accessible to the research team. In MMLA research, ethical considerations are paramount due to the potential risks associated with tracking individuals' detailed learning behaviours through sensor technologies in educational settings. These risks primarily revolve around privacy and data security concerns. Moreover, ensuring equitable and inclusive data collection and analysis is crucial as they directly impact fairness in data and modelling. Therefore, a rigorous ethical framework emphasizing privacy protection, data security, and equitable practices is essential to safeguard the rights and autonomy of participants and maintain the integrity of MMLA research.</p> <ref id="AN0178994769-43"> <title> REFERENCES </title> <blist> <bibl id="bib1" idref="ref15" type="bt">1</bibl> <bibtext> Alibali, M. W., & Nathan, M. J. (2007). Teachers' gestures as a means of scaffolding students' understanding: Evidence from an early algebra lesson. In Goldman, R., Pea, R., Barron, B., & Derry, S. J. (Eds.). Video research in the learning sciences (pp. 349 – 365). Routledge. https://doi.org/10.1111/j.1467‐8535.2008.00890_7.x</bibtext> </blist> <blist> <bibl id="bib2" idref="ref5" type="bt">2</bibl> <bibtext> Alibali, M. W., & Nathan, M. J. (2012). Embodiment in mathematics teaching and learning: Evidence from learners' and teachers' gestures. Journal of the Learning Sciences, 21 (2), 247 – 286. https://doi.org/10.1080/10508406.2011.611446</bibtext> </blist> <blist> <bibl id="bib3" idref="ref4" type="bt">3</bibl> <bibtext> Alibali, M. W., Nathan, M. J., Church, R. B., Wolfgram, M. S., Kim, S., & Knuth, E. J. (2013). Teachers' gestures and speech in mathematics lessons: Forging common ground by resolving trouble spots. ZDM ‐ International Journal on Mathematics Education, 45 (3), 425 – 440. https://doi.org/10.1007/s11858‐012‐0476‐0</bibtext> </blist> <blist> <bibl id="bib4" idref="ref8" type="bt">4</bibl> <bibtext> Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3 (2), 220 – 238.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref45" type="bt">5</bibl> <bibtext> Boulton, C. A., Kent, C., & Williams, H. T. P. (2018). Virtual learning environment engagement and learning outcomes at a 'bricks‐and‐mortar' university. Computers in Education, 126 (June), 129 – 142. https://doi.org/10.1016/j.compedu.2018.06.031</bibtext> </blist> <blist> <bibl id="bib6" idref="ref46" type="bt">6</bibl> <bibtext> Carini, R. M., Kuh, G. D., & Klein, S. P. (2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47 (1), 1 – 32. https://doi.org/10.1007/s11162‐005‐8150‐9</bibtext> </blist> <blist> <bibl id="bib7" idref="ref23" type="bt">7</bibl> <bibtext> Chu, M., & Kita, S. (2011). The nature of gestures' beneficial role in spatial problem solving. Journal of Experimental Psychology: General, 140 (1), 102 – 116. https://doi.org/10.1037/a0021790</bibtext> </blist> <blist> <bibl id="bib8" idref="ref28" type="bt">8</bibl> <bibtext> Chu, M., & Kita, S. (2016). Co‐thought and co‐speech gestures are generated by the same action generation process. Journal of Experimental Psychology. Learning, Memory, and Cognition, 42 (2), 257 – 270. https://doi.org/10.1037/xlm0000168</bibtext> </blist> <blist> <bibl id="bib9" idref="ref71" type="bt">9</bibl> <bibtext> Coolican, H. (2017). Research methods and statistics in psychology. Psychology Press.</bibtext> </blist> <blist> <bibtext> Davidsen, J., & Ryberg, T. (2017). "This is the size of one meter": Children's bodily‐material collaboration. International Journal of Computer‐Supported Collaborative Learning, 12 (1), 65 – 90. https://doi.org/10.1007/s11412‐017‐9248‐8</bibtext> </blist> <blist> <bibtext> Echeverría, V., Avendaño, A., Chiluiza, K., Vásquez, A., & Ochoa, X. (2014). Presentation skills estimation based on video and kinect data analysis. In MLA 2014 ‐ Proceedings of the 2014 ACM Multimodal Learning Analytics Workshop and Grand Challenge, Co‐Located with ICMI 2014 (pp. 53–60). https://doi.org/10.1145/2666633.2666641</bibtext> </blist> <blist> <bibtext> Edwards, L., Moore‐Russo, D., & Ferrara, F. (2014). Emerging perspectives on gesture and embodiment in mathematics. IAP.</bibtext> </blist> <blist> <bibtext> Elmoazen, R., Saqr, M., Tedre, M., & Hirsto, L. (2022). A systematic literature review of empirical research on epistemic network analysis in education. IEEE Access, 10, 17330 – 17348. https://doi.org/10.1109/ACCESS.2022.3149812</bibtext> </blist> <blist> <bibtext> Glaser, B. G., & Strauss, A. L. (2017). Discovery of grounded theory: Strategies for qualitative research. Routledge.</bibtext> </blist> <blist> <bibtext> Guo, Y., Li, Z., Li, Z., Du, X., Quan, S., & Xu, Y. (2022). PoP‐net: Pose over parts network for multi‐person 3D pose estimation from a depth image. In Proceedings ‐ 2022 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Vol. 2 (pp. 3917–3926). https://doi.org/10.1109/WACV51458.2022.00397</bibtext> </blist> <blist> <bibtext> Hassan, J., Leong, J., & Schneider, B. (2021). Multimodal data collection made easy: The EZ‐MMLA toolkit. ACM International Conference on Learning Analytics, LAK '21.</bibtext> </blist> <blist> <bibtext> Hegarty, M., Mayer, S., Kriz, S., & Keehner, M. (2005). The role of gestures in mental animation. Spatial Cognition and Computation, 5 (4), 333 – 356. https://doi.org/10.1207/s15427633scc0504_3</bibtext> </blist> <blist> <bibtext> Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture‐for‐conceptualization hypothesis. Psychological Review, 124 (3), 245 – 266. https://doi.org/10.1037/rev0000059</bibtext> </blist> <blist> <bibtext> Lee, O., & Fradd, S. H. (1998). Science for all, including students from non‐English‐language backgrounds. Educational Researcher, 27 (4), 12 – 21. https://doi.org/10.3102/0013189X027004012</bibtext> </blist> <blist> <bibtext> Lee, S., Shih, M., Liang, J., & Tseng, Y. (2021). Investigating learners' engagement and science learning outcomes in different designs of participatory simulated games. British Journal of Educational Technology, 52 (3), 1197 – 1214. https://doi.org/10.1111/bjet.13067</bibtext> </blist> <blist> <bibtext> Logan, T., Lowrie, T., & Diezmann, C. M. (2014). Co‐thought gestures: Supporting students to successfully navigate map tasks. Educational Studies in Mathematics, 87 (1), 87 – 102. https://doi.org/10.1007/s10649‐014‐9546‐2</bibtext> </blist> <blist> <bibtext> Marquart, C. L., Swiecki, Z., Eagan, B., & Shaffer, D. W. (2018). ncodeR: Techniques for automated classifiers. R Package Version 0.1.2. https://cran.r‐project.org/package=ncodeR</bibtext> </blist> <blist> <bibtext> Ochoa, X., Domínguez, F., Guamán, B., Maya, R., Falcones, G., & Castells, J. (2018). The RAP system: Automatic feedback of oral presentation skills using multimodal analysis and low‐cost sensors. In ACM International Conference Proceeding Series (pp. 360–364). https://doi.org/10.1145/3170358.3170406</bibtext> </blist> <blist> <bibtext> Schneider, B., & Blikstein, P. (2015). Unraveling students' interaction around a tangible interface using multimodal learning analytics. Journal of Educational Data Mining, 7 (3), 89 – 116.</bibtext> </blist> <blist> <bibtext> Schwartz, D. L., & Black, J. B. (1996). Shuttling between depictive models and abstract rules: Induction and fallback. Cognitive Science, 20 (4), 457 – 497.</bibtext> </blist> <blist> <bibtext> Shaffer, D. W. (2017). Quantitative ethnography. Cathcart Press.</bibtext> </blist> <blist> <bibtext> Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics, 3 (3), 9 – 45.</bibtext> </blist> <blist> <bibtext> Si, J., Lin, J., Jiang, F., & Shen, R. (2019). Hand‐raising gesture detection in real classrooms using improved R‐FCN. Neurocomputing, 359, 69 – 76. https://doi.org/10.1016/j.neucom.2019.05.031</bibtext> </blist> <blist> <bibtext> Sung, H., Shengyang, C., Ruis, A. R., & Shaffer, D. W. (2019). Reading for breadth, reading for depth: Understanding the relationship between reading and complex thinking using epistemic network analysis. In K. Lund, G. P. Niccolai, E. Lavoué, C. Hmelo‐Silver, G. Gweon, & M. Baker (Eds.), A Wide Lens: Combining Embodied, Enactive, Extended, and Embedded Learning in Collaborative Settings, 13th International Conference on Computer Supported Collaborative Learning (CSCL) 2019 (Vol. 1, pp. 376 – 383). International Society of the Learning Sciences. https://repository.isls.org//handle/1/4428</bibtext> </blist> <blist> <bibtext> Sung, H., Swart, M., & Nathan, M. J. (2021). Enhancing K‐12 pre‐service teachers' embodied understanding of the geometry knowledge through online collaborative design. In D. Olanoff, K. Johnson, & S. M. Spitzer (Eds.), Proceedings of the forty‐third annual meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education (pp. 909 – 917). Philadelphia, PA. https://<ulink href="http://www.pmena.org/pmenaproceedings/PMENA%2043%202021%20Proceedings.pdf">www.pmena.org/pmenaproceedings/PMENA%2043%202021%20Proceedings.pdf</ulink></bibtext> </blist> <blist> <bibtext> Sung, H., Swart, M. I., & Nathan, M. J. (2022). Methods for analyzing temporally entangled multimodal data. In A. Weinberger, W. Chen, D. Hernández‐Leo, & B. Chen (Eds.), Proceedings of the 15th International Conference on Computer Supported Collaborative Learning ‐ CSCL 2022 (pp. 242 – 249). International Society of the Learning Sciences. https://repository.isls.org//handle/1/8282</bibtext> </blist> <blist> <bibtext> Suthers, D. D. (2006). Technology affordances for intersubjective meaning making: A research agenda for CSCL. International Journal of Computer‐Supported Collaborative Learning, 1 (3), 315 – 337. https://doi.org/10.1007/s11412‐006‐9660‐y</bibtext> </blist> <blist> <bibtext> Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.</bibtext> </blist> <blist> <bibtext> Walkington, C., Chelule, G., Woods, D., & Nathan, M. J. (2019). Collaborative gesture as a case of extended mathematical cognition. Journal of Mathematical Behavior, 55 (March), 100683. https://doi.org/10.1016/j.jmathb.2018.12.002</bibtext> </blist> <blist> <bibtext> Wang, M., Tighe, J., & Modolo, D. (2020). Combining detection and tracking for human pose estimation in videos. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 11085–11093). https://doi.org/10.1109/CVPR42600.2020.01110</bibtext> </blist> <blist> <bibtext> Wertsch, J. V. (1993). Voices of the mind. Harvard University Press.</bibtext> </blist> <blist> <bibtext> Zurina, H., & Williams, J. (2011). Gesturing for oneself. Educational Studies in Mathematics, 77 (2–3), 175 – 188. https://doi.org/10.1007/s10649‐010‐9294‐x</bibtext> </blist> </ref> <aug> <p>By Hanall Sung and Mitchell J. Nathan</p> <p>Reported by Author; Author</p> </aug> <nolink nlid="nl1" bibid="bib32" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib33" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib36" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib34" firstref="ref6"></nolink> <nolink nlid="nl5" bibid="bib19" firstref="ref7"></nolink> <nolink nlid="nl6" bibid="bib11" firstref="ref9"></nolink> <nolink nlid="nl7" bibid="bib23" firstref="ref10"></nolink> <nolink nlid="nl8" bibid="bib24" firstref="ref11"></nolink> <nolink nlid="nl9" bibid="bib10" firstref="ref17"></nolink> <nolink nlid="nl10" bibid="bib12" firstref="ref18"></nolink> <nolink nlid="nl11" bibid="bib18" firstref="ref19"></nolink> <nolink nlid="nl12" bibid="bib17" firstref="ref20"></nolink> <nolink nlid="nl13" bibid="bib25" firstref="ref22"></nolink> <nolink nlid="nl14" bibid="bib37" firstref="ref25"></nolink> <nolink nlid="nl15" bibid="bib21" firstref="ref31"></nolink> <nolink nlid="nl16" bibid="bib28" firstref="ref38"></nolink> <nolink nlid="nl17" bibid="bib35" firstref="ref39"></nolink> <nolink nlid="nl18" bibid="bib20" firstref="ref47"></nolink> <nolink nlid="nl19" bibid="bib27" firstref="ref48"></nolink> <nolink nlid="nl20" bibid="bib29" firstref="ref49"></nolink> <nolink nlid="nl21" bibid="bib31" firstref="ref50"></nolink> <nolink nlid="nl22" bibid="bib30" firstref="ref51"></nolink> <nolink nlid="nl23" bibid="bib16" firstref="ref58"></nolink> <nolink nlid="nl24" bibid="bib15" firstref="ref59"></nolink> <nolink nlid="nl25" bibid="bib22" firstref="ref63"></nolink> <nolink nlid="nl26" bibid="bib14" firstref="ref65"></nolink> <nolink nlid="nl27" bibid="bib26" firstref="ref66"></nolink> <nolink nlid="nl28" bibid="bib97" firstref="ref70"></nolink> <nolink nlid="nl29" bibid="bib13" firstref="ref78"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1434977
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Your Body Tells How You Engage in Collaboration: Machine-Detected Body Movements as Indicators of Engagement in Collaborative Math Knowledge Building
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Hanall+Sung%22">Hanall Sung</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9076-3762">0000-0001-9076-3762</externalLink>)<br /><searchLink fieldCode="AR" term="%22Mitchell+J%2E+Nathan%22">Mitchell J. Nathan</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22British+Journal+of+Educational+Technology%22"><i>British Journal of Educational Technology</i></searchLink>. 2024 55(5):1950-1973.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 24
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Cooperative+Learning%22">Cooperative Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Motion%22">Motion</searchLink><br /><searchLink fieldCode="DE" term="%22Human+Body%22">Human Body</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Analytics%22">Learning Analytics</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Processes%22">Learning Processes</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Education%22">Mathematics Education</searchLink><br /><searchLink fieldCode="DE" term="%22Learner+Engagement%22">Learner Engagement</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+Secondary+Education%22">Elementary Secondary Education</searchLink><br /><searchLink fieldCode="DE" term="%22Preservice+Teachers%22">Preservice Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Geometry%22">Geometry</searchLink><br /><searchLink fieldCode="DE" term="%22Gamification%22">Gamification</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Simulation%22">Computer Simulation</searchLink><br /><searchLink fieldCode="DE" term="%22Nonverbal+Learning%22">Nonverbal Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Verbal+Learning%22">Verbal Learning</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/bjet.13473
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0007-1013<br />1467-8535
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Collaborative learning, driven by knowledge co-construction and meaning negotiation, is a pivotal aspect of educational contexts. While gesture's importance in conveying shared meaning is recognized, its role in collaborative group settings remains understudied. This gap hinders accurate and equitable assessment and instruction, particularly for linguistically diverse students. Advancements in multimodal learning analytics, leveraging sensor technologies, offer innovative solutions for capturing and analysing body movements. This study employs these novel approaches to demonstrate how learners' machine-detected body movements during the learning process relate to their verbal and nonverbal contributions to the co-construction of embodied math knowledge. These findings substantiate the feasibility of utilizing learners' machine-detected body movements as a valid indicator for inferring their engagement with the collaborative knowledge construction process. In addition, we empirically validate that these inferred different levels of learner engagement indeed impact the desired learning outcomes of the intervention. This study contributes to our scientific understanding of multimodal approaches to knowledge expression and assessment in learning, teaching, and collaboration.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1434977
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1434977
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/bjet.13473
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 24
        StartPage: 1950
    Subjects:
      – SubjectFull: Cooperative Learning
        Type: general
      – SubjectFull: Motion
        Type: general
      – SubjectFull: Human Body
        Type: general
      – SubjectFull: Learning Analytics
        Type: general
      – SubjectFull: Learning Processes
        Type: general
      – SubjectFull: Mathematics Education
        Type: general
      – SubjectFull: Learner Engagement
        Type: general
      – SubjectFull: Elementary Secondary Education
        Type: general
      – SubjectFull: Preservice Teachers
        Type: general
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Geometry
        Type: general
      – SubjectFull: Gamification
        Type: general
      – SubjectFull: Computer Simulation
        Type: general
      – SubjectFull: Nonverbal Learning
        Type: general
      – SubjectFull: Verbal Learning
        Type: general
    Titles:
      – TitleFull: Your Body Tells How You Engage in Collaboration: Machine-Detected Body Movements as Indicators of Engagement in Collaborative Math Knowledge Building
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Hanall Sung
      – PersonEntity:
          Name:
            NameFull: Mitchell J. Nathan
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 0007-1013
            – Type: issn-electronic
              Value: 1467-8535
          Numbering:
            – Type: volume
              Value: 55
            – Type: issue
              Value: 5
          Titles:
            – TitleFull: British Journal of Educational Technology
              Type: main
ResultId 1