View in EDS HTML Full Text PDF Full Text

Unraveling Temporally Entangled Multimodal Interactions: Investigating Verbal and Nonverbal Contributions to Collaborative Construction of Embodied Math Knowledge

Saved in:
Bibliographic Details
Title: Unraveling Temporally Entangled Multimodal Interactions: Investigating Verbal and Nonverbal Contributions to Collaborative Construction of Embodied Math Knowledge
Language: English
Authors: Hanall Sung (ORCID 0000-0001-9076-3762), Mitchell J. Nathan
Source: International Journal of Educational Technology in Higher Education. 2025 22.
Availability: BioMed Central, Ltd. Available from: Springer Nature. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: customerservice@springernature.com; Web site: https://www.springer.com/gp/biomedical-sciences
Peer Reviewed: Y
Page Count: 28
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Technology Uses in Education, Epistemology, Research and Development, Nonverbal Learning, Verbal Learning, Learning Modalities, Undergraduate Students, Mathematics Education, Education Majors, Universities, Cooperative Learning, Intermode Differences, Learning Analytics, Constructivism (Learning)
DOI: 10.1186/s41239-025-00504-6
ISSN: 2365-9440
Abstract: In various technology-enhanced learning (TEL) environments, knowledge co-creation progresses through multimodal interactions that integrate verbal and nonverbal modalities, such as speech and gestures. This study investigated two distinct analytical approaches for analyzing multimodal interactions--triangulating and interleaving--by applying them to collaborative learning processes during an online embodied mathematics intervention. The findings demonstrate that the interleaving approach captures the temporal dynamics and nuanced interplay between multimodal events, providing deeper insights into how shared meaning-making evolves over time. In contrast, the triangulating approach effectively identifies cumulative interaction patterns but does not account for their temporal structure. Specifically, the interleaving approach, employing epistemic network analysis, revealed statistically significant differences in discourse patterns between learners with larger and smaller variances in upper body movements during the co-design activity. These findings underscore the complementary value of the interleaving approach in analyzing multimodal interactions and offer practical implications for advancing understanding of collaborative learning processes in TEL environments.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1461019
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGJBm4F6z5WyuKUI7TaYyRxAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDE4voT0c_st7AliYWQIBEICBm-_Zag6gKG9_SorMqtt4hkHUlOxRcW60pnUB7zW8W-GxiQQSiZihXZbovguiuosOcESxNOjubpmOHgKOqZloR1YjLCGzAJkidlNCqJTlcDiAFjndXQmB1p3oN5bmiUmonSlCs9JMpu1w4rilFiO7b1fS2DriRD5BJkpuYRYpftJwzV7MkQr2xtoq4yWoW81JU2YqjV2xkotuk79W
Text:
  Availability: 1
  Value: <anid>AN0183073972;[jq69]13feb.25;2025Feb19.02:54;v2.2.500</anid> <title id="AN0183073972-1">Unraveling temporally entangled multimodal interactions: investigating verbal and nonverbal contributions to collaborative construction of embodied math knowledge </title> <p>In various technology-enhanced learning (TEL) environments, knowledge co-creation progresses through multimodal interactions that integrate verbal and nonverbal modalities, such as speech and gestures. This study investigated two distinct analytical approaches for analyzing multimodal interactions—triangulating and interleaving—by applying them to collaborative learning processes during an online embodied mathematics intervention. The findings demonstrate that the interleaving approach captures the temporal dynamics and nuanced interplay between multimodal events, providing deeper insights into how shared meaning-making evolves over time. In contrast, the triangulating approach effectively identifies cumulative interaction patterns but does not account for their temporal structure. Specifically, the interleaving approach, employing epistemic network analysis, revealed statistically significant differences in discourse patterns between learners with larger and smaller variances in upper body movements during the co-design activity. These findings underscore the complementary value of the interleaving approach in analyzing multimodal interactions and offer practical implications for advancing understanding of collaborative learning processes in TEL environments.</p> <p>Keywords: Multimodal learning analytics; Temporal; Learning process; Body movement; Gesture; Motion sensing; Embodied learning; Epistemic network analysis; Collaborative learning; Psychology and Cognitive Sciences Psychology</p> <hd id="AN0183073972-2">Introduction</hd> <p>Peer learning processes in various technology-enhanced learning (TEL) environments have evolved into more complex and interactive forms. Within these environments, the co-creation of knowledge among individuals progresses through <emph>multimodal interactions</emph>, encompassing verbal and nonverbal interactions such as speech, gestures, and digital tool utilization. Understanding peer learning processes within such contexts necessitates a thorough examination of the primary modalities utilized for communication during collaborative knowledge construction. Recent advancements in technological innovations, including sensor technologies and artificial intelligence tools, enable researchers in multimodal learning analytics (MMLA; Blikstein, [<reflink idref="bib4" id="ref1">4</reflink>]) to gather vast amounts of data on multimodal interactions. For instance, MMLA researchers have explored automated detection of body synchronization, movement, and positioning, alongside speech, to broaden our understanding of collaborative peer learning processes (e.g., Huang et al., [<reflink idref="bib17" id="ref2">17</reflink>]; Schneider & Blikstein, [<reflink idref="bib27" id="ref3">27</reflink>]). However, we still encounter methodological challenges in accurately modeling and analyzing multimodal interactions to improve our understanding of collaborative knowledge building and sharing.</p> <p>One challenge derives from the temporal dynamics inherent in multimodal interactions during peer learning processes, which we term as "temporally entangled multimodal interactions" (Sung et al., [<reflink idref="bib35" id="ref4">35</reflink>]). Events in different modalities often contemporaneously occur in specific sequences, dynamically contributing to the overall collaborative meaning-making central to peer learning processes. Despite of the importance of unraveling these temporally entangled multimodal interactions in understanding when, how, and what one learns (Nathan, [<reflink idref="bib21" id="ref5">21</reflink>]), the prevailing approach in MMLA research is <emph>triangulation</emph>. In this approach, independent modalities' accumulative accounts are typically used and compared in a synchronic (i.e., static) manner to assess how they separately or complementarily describe the same collaborative meaning-making process (e.g., Noroozi et al., [<reflink idref="bib22" id="ref6">22</reflink>]; Starr et al., [<reflink idref="bib32" id="ref7">32</reflink>]). Triangulation is useful for validating novel proxies or metrics that can be automatically tracked through sensors or tools by cross-referencing with human observations or established measures. For example, Noroozi et al. ([<reflink idref="bib22" id="ref8">22</reflink>]) exemplify the triangulating approach by integrating multiple data sources to validate novel proxies for understanding learners' self-regulated learning behaviors. The study utilized multimodal data, including physiological signals, facial expressions, and computer interaction logs, to design visual learning analytics for analyzing regulation processes. These automatically tracked metrics were triangulated with traditional self-report questionnaires and human observations to ensure validity and reliability. By combining multiple data sources, Noroozi et al. demonstrated how triangulating different modalities provides a more comprehensive and reliable understanding of learners' regulatory processes, enhancing the validity of sensor-based measures. However, the triangulating approach often overlooks the temporal sequences of events, as it relies on correlating accumulative accounts of different modalities. Consequently, it may inadvertently omit crucial contextual information about peer learning processes (Sung et al., [<reflink idref="bib35" id="ref9">35</reflink>]).</p> <p>As an alternative way to address this analytic need, researchers can employ the <emph>interleaving</emph> approach to explore the temporal interplay between the events that are registered in different modalities (Sung et al., [<reflink idref="bib35" id="ref10">35</reflink>]). Unlike the triangulating approach, the interleaving approach allows the analyst to grasp how the meaning-making arises through the dynamic interaction of multiple modalities in a diachronic manner. By leveraging both the timing and the sequential order of multimodal events during collaborative learning, this approach enhance our understanding of how shared meaning-making occurs (e.g., Bridges et al., [<reflink idref="bib5" id="ref11">5</reflink>]). For example, Bridges et al. ([<reflink idref="bib5" id="ref12">5</reflink>]) conducted a micro-ethnographic analysis of classroom discourse during problem-based learning activities with medical students. By examining the moment-by-moment interplay between modes such as speech, gaze, and posture, they documented how events in one mode (e.g., speech) were sequentially interleaved with those in other modes (e.g., gaze or posture). Their analysis revealed intricate interrelationships among multimodal events and highlighted their impact on the collaborative learning process. However, the task of disentangling the temporal sequences of multimodal events is not trivial. It requires extensive time and effort to meticulously trace moment-by-moment events using micro-ethnographic analyses, as well as demanding analyses and interpretations that reveal holistic meaning (Sung et al., [<reflink idref="bib35" id="ref13">35</reflink>]). To address these challenges, Sung et al. ([<reflink idref="bib35" id="ref14">35</reflink>]) have introduced a novel method of implementing the interleaving approach, which integrates MMLA techniques with discourse analysis tools, specifically utilizing epistemic network analysis (ENA; Shaffer et al., [<reflink idref="bib30" id="ref15">30</reflink>]). This analytical approach offers systematic procedures for disentangling temporally entangled multimodal interactions, with the aim of reducing labor intensity, increasing reliability, and facilitating scalability.</p> <p>In this paper, we employed both triangulating and interleaving approaches to analyze a peer learning process, presenting empirical findings from each analysis of multimodal interactions and examine the affordances and challenges of both approaches. We conducted an online action-based learning intervention for college students majoring in math education (i.e., K-12 pre-service math teachers), providing them with an opportunity to engage in embodied mathematical thinking and reasoning. The intervention involved a co-design activity where learners collectively discussed and formulated actions to enhance students' understanding of mathematical concepts in geometry classrooms. To investigate peer learning processes, we examined how each learner contributed verbally and nonverbally to the collaborative construction of embodied mathematical knowledge. Specifically, we utilized automatic detection of variances in learners' body movements as a proxy to infer their verbal and nonverbal contributions during the peer learning process. By applying both triangulating and interleaving approaches separately, we investigated how learners' machine-detected body movements relate to their verbal and nonverbal contributions to co-creation of embodied math knowledge. Our investigation aims to empirically illustrate how the interleaving approach reveals aspects of collaborative meaning-making processes that are not readily obtained using the triangulating approach, thus highlighting its complementary role in multimodal analysis of peer learning in TEL environments.</p> <hd id="AN0183073972-3">Theoretical background</hd> <p></p> <hd id="AN0183073972-4">Collaborative gesture</hd> <p>Collaborative learning involves the co-construction of knowledge and the negotiation of meaning among individuals (Suthers, [<reflink idref="bib36" id="ref16">36</reflink>]; Vygotsky, [<reflink idref="bib37" id="ref17">37</reflink>]; Wertsch, [<reflink idref="bib39" id="ref18">39</reflink>]). Across the disciplines, individuals commonly use hand gestures and whole-body movements alongside speech to articulate their thoughts and convey meanings (e.g., Alibali & Nathan, [<reflink idref="bib1" id="ref19">1</reflink>], [<reflink idref="bib2" id="ref20">2</reflink>]; [<reflink idref="bib33" id="ref21">33</reflink>]Davidsen & Ryberg, [<reflink idref="bib8" id="ref22">8</reflink>]; Sung et al., [<reflink idref="bib34" id="ref23">34</reflink>]). Many multimodal analysis studies, which aim to identify gestures associated with meaning-making, primarily focus on <emph>representational gestures</emph> that "depict action, motion, or shape, or that indicate location or trajectory" (Kita et al., [<reflink idref="bib14" id="ref24">14</reflink>], p. 245), henceforth referred to simply as gestures.</p> <p>Existing research has highlighted the significance of gesture in establishing common ground and reflecting shared meaning-making (e.g., Alibali & Nathan, [<reflink idref="bib2" id="ref25">2</reflink>]; Alibali et al., [<reflink idref="bib3" id="ref26">3</reflink>]; Schneider et al., [<reflink idref="bib26" id="ref27">26</reflink>]; Walkington et al., [<reflink idref="bib38" id="ref28">38</reflink>]). For example, Schneider et al. ([<reflink idref="bib26" id="ref29">26</reflink>]) explored the role of gesture and gaze in dyadic interactions during collaborative learning, emphasizing the significance of nonverbal interactions for understanding peer learning processes. By analyzing how nonverbal interactions like gaze direction and gestures contribute to joint attention and shared understanding, the authors highlighted the importance of integrating these modalities into studies of collaborative learning. Their work demonstrates how multimodal approaches can uncover the subtle dynamics of coordination, offering valuable insights into the mechanisms underlying effective peer collaboration. While studies such as Schneider et al. ([<reflink idref="bib26" id="ref30">26</reflink>]) have explored the role of gestures in peer learning processes, particularly in dyadic interactions, much of the existing literature has focused on individual rather than group settings and has not specifically addressed the role of gestures as a tool for collaboration and collective cognitive processes. By investigating how individuals use gesture in collaborative learning contexts, we can deepen our understanding of how they contribute verbally and nonverbally to the co-construction of knowledge, as well as how they interpret the shared meaning conveyed through various modalities of communication among peers.</p> <p>As a way to identify gesture use during collaboration, Walkington et al. ([<reflink idref="bib38" id="ref31">38</reflink>]) defined <emph>collaborative gesture</emph> as "gestural exchanges that take place as learners discuss and explore mathematical ideas, using their bodies in concert to accomplish a shared goal" (p. 100,783). The authors suggested that learners use different forms of collaborative gestures to extend mathematical ideas over multiple bodies of the group members as they explore, refine, and extend each other's embodied forms of mathematical reasoning. Walkington and colleagues then classified these collaborative gestures into 4 main categories: (<reflink idref="bib1" id="ref32">1</reflink>) <emph>Echoing</emph> (one learner producing the same or very similar gesture after the initial original gesture produced by another learner), (<reflink idref="bib2" id="ref33">2</reflink>) <emph>Mirroring</emph> (one learner producing the same or very similar gesture nearly at the same time when the initial original gesture produced by another learner), (<reflink idref="bib3" id="ref34">3</reflink>) <emph>Alternating</emph> (one learner producing follow-up gesture building upon or altering the initial original gesture produced by another learner), and (<reflink idref="bib4" id="ref35">4</reflink>) <emph>Joint</emph> (multiple learners producing a unified set of gesture by manipulating, pointing to, or formulating mathematical objects). If a learner's representational gesture fit into any of the categories, it is considered as collaborative gesture.</p> <p>The framework of collaborative gesture proposed by Walkington et al. ([<reflink idref="bib38" id="ref36">38</reflink>]) was initially designed to capture the collaborative use of gestures within physically co-located collaborative learning environments. However, there is a growing necessity to extend this framework to encompass broader learning contexts, such as online learning environments, which are increasingly prevalent. In our research, we employ this framework to analyze learners' gesture use during collaborative learning within an online learning setting. To the best of our knowledge, this represents the first application of the framework to virtual learning environments, thereby expanding our understanding of multimodal ways of collaborative knowledge building and sharing across various TEL environments.</p> <hd id="AN0183073972-5">Co-thought gesture</hd> <p>Another aspect to consider when examining the use of gesture in collaboration is whether the gesture is accompanied by speech or silent thought. Individuals naturally gesture not only while speaking or describing problem-solving processes (<emph>co-speech gesture</emph>) but also during silent problem-solving or thought processes (Hegarty et al., [<reflink idref="bib12" id="ref37">12</reflink>]; Kita et al., [<reflink idref="bib14" id="ref38">14</reflink>]; Schwartz & Black, [<reflink idref="bib28" id="ref39">28</reflink>]). Chu and Kita ([<reflink idref="bib6" id="ref40">6</reflink>]) have categorized such silent, non-communicative gestures produced during problem-solving situations as <emph>co-thought gesture.</emph> These co-thought gestures occur when individuals face challenging problem-solving tasks (Chu & Kita, [<reflink idref="bib6" id="ref41">6</reflink>]) or when they reflect on their embodied thoughts (Zurina & Williams, [<reflink idref="bib40" id="ref42">40</reflink>]). For example, Zurina and Williams ([<reflink idref="bib40" id="ref43">40</reflink>]) discovered that students produced co-thought gestures when encountering difficulties during group work on fraction problems. Despite being engaged in a collaborative activity, when students needed to introspect on their own thoughts, they diverted their attention from the group and made small-sized, silent co-thought gestures related to the tasks. These gestures were not meant to directly contribute to group discussions, as they were not accompanied by speech; rather, their purpose was to clarify the students' individual thoughts and aid in grasping the mathematical task at hand. Zurina and Williams suggested that these "gestures for oneself" were prompted by the necessity to enhance the visual-spatial and enactive aspects of internal thoughts. Considering that the function of co-thought gestures in collaborative settings differs from that of co-speech gestures, it is essential to recognize these gestures as distinct phenomena, despite both involving representational gestures (Kita et al., [<reflink idref="bib14" id="ref44">14</reflink>]).</p> <p>The phenomenon of co-thought gesture remains considerably less understood compared to co-speech gesture, and its underlying production mechanism remains largely unexplored. Consequently, there remains a substantial amount yet to be revealed regarding the precise functions of co-thought gestures within collaborative learning contexts—how individuals produce co-thought gestures to echo, mirror, or alternate peers' ideas. Previous studies have suggested that co-thought and co-speech gestures may stem from a shared mechanism that activates spatio-motoric information (Chu & Kita, [<reflink idref="bib6" id="ref45">6</reflink>], [<reflink idref="bib7" id="ref46">7</reflink>]; Kita et al., [<reflink idref="bib14" id="ref47">14</reflink>]). One indication of this is that individuals who produce more co-thought gestures also tend to produce more co-speech gestures (Chu & Kita, [<reflink idref="bib7" id="ref48">7</reflink>]). Another common empirical feature observed in co-thought gestures is that the hand or arm movements tend to be much smaller and closer to the gesturer's body compared to those used in interpersonal communication contexts (Logan et al., [<reflink idref="bib16" id="ref49">16</reflink>]; Zurina & Williams, [<reflink idref="bib40" id="ref50">40</reflink>]). This observation suggests the potential utility of employing MMLA techniques to identify co-speech and co-thought gestures through automatic detection of body movements using computer vision tools.</p> <hd id="AN0183073972-6">Automatic detection of body movements</hd> <p>Recent technological advancements have enabled the automatic capture and detection of body movements at high frequencies (e.g., 30 frames per second) and with fine-grained levels of detail (e.g., tracking dozens of body landmarks). This technology provides researchers with innovative opportunities to examine how individuals utilize their bodies during collaboration. MMLA researchers employ Kinect sensors or computer vision tools to collect and analyze skeletal body joint data for the automatic detection of body movements or gestures (e.g., Ochoa et al., [<reflink idref="bib24" id="ref51">24</reflink>]; Schneider & Blikstein, [<reflink idref="bib27" id="ref52">27</reflink>]; Starr et al., [<reflink idref="bib32" id="ref53">32</reflink>]) For example, Starr et al. ([<reflink idref="bib32" id="ref54">32</reflink>]) conducted analyses on the total quantity of movement across upper body joints and body parts captured by a Kinect sensor, as well as the overall amount of verbal communication. The researchers argue that the quantity of body movements and verbal exchanges significantly correlate with the quality of collaboration, indicating these metrics can serve as proxy measures for effective collaborative peer interactions. In a similar vein, Schneider and Blikstein ([<reflink idref="bib27" id="ref55">27</reflink>]) quantified the quantity of hand movements exhibited by student dyads and categorized students into either active or passive roles based on this measure. They asserted that students' leadership during problem-solving activities can be distinguished by examining their hand movements, and their roles are associated with learning gains; taking an active role positively correlates with learning gains, whereas adopting a passive role negatively correlates with learning gains. These studies suggest that the quantity of body movements exhibited during peer learning processes, as automatically detected through upper body joint locations, could offer novel insights into collaborative learning.</p> <p>Expanding on previous research in the field of MMLA, Sung & Nathan ([<reflink idref="bib33" id="ref56">33</reflink>]) proposed a novel method for automatically detecting upper body movements through the use of <emph>bounding boxes</emph> derived from body joint data. Employing a web-based motion-sensing tool developed with computer vision systems, we captured numerical location data of an individual's upper body joints, including the x- and y-coordinates of the left and right shoulders, elbows, and wrists. This data was then used to construct a bounding box around the upper body, representing the smallest enclosing box around the object (in this case, upper body). This method enables the monitoring of changes in body joint positions across both temporal and spatial dimensions. Thus, variances in upper body movements could be measured by tracking changes in the dimensions of the bounding boxes (for more details regarding bounding boxes, see Sung & Nathan, [<reflink idref="bib33" id="ref57">33</reflink>]).</p> <p>In our prior study (Sung & Nathan, [<reflink idref="bib33" id="ref58">33</reflink>]), we showed that learners who exhibited larger variance in upper body movements during a collaborative learning task demonstrated higher levels of engagement, as indicated by their increased production of both contextually relevant gestures and verbal utterances. This suggests that machine-detected body movements can serve as a valid proxy for inferring learners' engagement in embodied learning activities. This prior study serves as a foundation for the current research and also suggests avenues for expansion. For one, the earlier work focused solely on whether gestures were accompanied by speech (co-speech gestures) or not (co-thought gestures), without exploring the collaborative functions of gestures, such as echoing, mirroring, or alternating others' embodied ideas, or initiating one's own ideas. Furthermore, the prior study demonstrated how differences in learner engagement, inferred from variances in upper body movements, directly impacted intervention outcomes. However, it did not thoroughly investigate the temporal dynamics of the peer learning process, particularly how multimodal events temporally and contextually interact during collaborative meaning-making. Building upon this prior work, our current study examines peer learning processes by examining how learner with larger or smaller variances of upper body movements exhibit distinct discourse patterns of verbal and gestural contributions to the co-construction of embodied mathematical knowledge, considering the temporal interplay between multimodal events.</p> <hd id="AN0183073972-7">Modeling temporality inherent in peer learning processes: multimodal matrix and epistemic net...</hd> <p>Learning is a dynamic process that unfolds gradually over time and is inherently cumulative (Nathan, [<reflink idref="bib20" id="ref59">20</reflink>]; Reimann, [<reflink idref="bib25" id="ref60">25</reflink>]). This cumulative nature implies that the sequence of events over the course of students' learning reflects both <emph>how</emph> and <emph>what</emph> one learns (Reimann, [<reflink idref="bib25" id="ref61">25</reflink>]). In TEL environments, where learning occurs through multimodal interactions, various events in modalities often co-occur temporally in specific sequences, influencing the process of shared meaning-making. Typical approaches, however, often focus on comparing and correlating cumulative accounts of interactions (Knight et al., [<reflink idref="bib15" id="ref62">15</reflink>]), overlooking temporal aspects and thus omitting contextual information of the process. Excluding such information from an analysis may impede truly capturing the most accurate understanding of the learning process (Mercer, [<reflink idref="bib18" id="ref63">18</reflink>]).</p> <p>To accurately model the temporal sequences of events in multimodal interactions during collaborative learning, researchers often face challenges in representing temporality within the form of transcripts. Evans et al. ([<reflink idref="bib10" id="ref64">10</reflink>]) noted that the way people perceive temporality in collaborative multimodal interactions is heavily influenced by how it is represented in the transcript. Linear transcripts often sacrifice the accurate representation of temporal nuances across different modalities (e.g., the duration of a verbal utterance compared to a gesture) and the temporal entanglements between multimodal events for the sake of readability. To address this issue, some researchers leverage human annotation tools designed to support transcribing multimodal interactions, such as ChronoViz, VIT, or V-note (Mitri et al., [<reflink idref="bib19" id="ref65">19</reflink>]). These multimodal annotation tools enable researchers to visualize multimodal events, make multiple data streams temporally interleave one another, and add customized annotations. Even after synchronization and integration of multimodal events, the task of modeling the temporal sequences of multimodal events remains challenging, as it entails extensive human effort to meticulously trace moment-by-moment occurrences through micro-ethnographic analyses (Sung et al., [<reflink idref="bib35" id="ref66">35</reflink>]). This process demands sophisticated analyses and interpretations to uncover holistic meaning from the data, which can be labor-intensive and time-consuming. To tackle this challenge, Sung et al. ([<reflink idref="bib35" id="ref67">35</reflink>]) suggested a more systematic way to apply the interleaving approach in analyzing multimodal interactions. In our previous research, we illustrated methods for operationalizing multimodal data and interpret meaning across multiple modalities in a more intuitive manner, all while taking into account the temporal structure of events. This was achieved with the aid of discourse analysis techniques and tools.</p> <p>Our approach, inspired by Echeverría et al. ([<reflink idref="bib9" id="ref68">9</reflink>]), is grounded in the methods of <emph>quantitative ethnography</emph> (Shaffer, [<reflink idref="bib29" id="ref69">29</reflink>]), which integrate qualitative and quantitative analysis. To structure and model multimodal data for analysis using the systematic ways of interleaving approach, we first employ the concept of the <emph>multimodal matrix</emph>, a conceptual data representation suggested by Echeverría et al. ([<reflink idref="bib9" id="ref70">9</reflink>]) (see Fig. 1). This method draws inspiration from the methodologies of quantitative ethnography (Shaffer, [<reflink idref="bib29" id="ref71">29</reflink>]). Here, we discuss the specifics of the multimodal matrix to demonstrate how we propose operationalizing temporally entangled multimodal interactions.</p> <p>Graph: Fig. 1 An exemplary representation of the multimodal matrix inspired by Echeverría et al. ([<reflink idref="bib9" id="ref72">9</reflink>])</p> <p>Figure 1 serves as an exemplary representation of the multimodal matrix used in this paper. Each type of multimodal data can be coded into <emph>multimodal observations</emph> (i.e., kinds of information derived from multimodal data; codes) that identify the <emph>dimensions of discourse</emph> (e.g., verbal, gestural, and potentially others). This means that different dimensions of multimodal discourse and their respective multimodal observations can be structured and analyzed together by tabulating a multimodal matrix. As shown in Fig. 1, each dimension of discourse can include several multimodal observations in a series of columns, where each column represents a different code. If there are three different epistemic codes for verbal dimension, then there is a different column for each to be scored. Each row in the matrix represents <emph>segments</emph>, the smallest unit of meaning considered for analysis (Shaffer, [<reflink idref="bib29" id="ref73">29</reflink>]). In each segment, a value of 1 indicates the presence of a feature within a given segment, and a value of 0 indicates its absence. Time windows (i.e., analytic time unit) for each segmentation can be pre-defined but can be segmented by the occurrence of natural pauses in speech or gesture production (e.g., Sung et al., [<reflink idref="bib35" id="ref74">35</reflink>];Sung & Nathan, [<reflink idref="bib33" id="ref75">33</reflink>]). Next, these segments grouped into <emph>stanzas</emph>, representing collections of rows considered to be within the same recent temporal context.</p> <p>Subsequently, epistemic network models can be constructed from the multimodal matrix using ENA, a discourse analysis technique for identifying and quantifying the connections among cognitive elements in a discourse (Shaffer et al., [<reflink idref="bib30" id="ref76">30</reflink>]). ENA generates dynamic nodal networks of discourse centered around a computed mean centroid, weighting interconnections between codes (Shaffer, [<reflink idref="bib29" id="ref77">29</reflink>]). The codes in ENA models correspond to the epistemic elements characterizing the discourse, with network edges representing the relative frequency of co-occurrence between two codes. Notably, the application of the moving stanza windows (Siebert-Evenstone et al., [<reflink idref="bib31" id="ref78">31</reflink>]) to construct a network model enables the capture of natural interactions between multimodal observations occurring within recent temporal contexts. Through this method, we can elucidate interactions between temporally and contextually entangled multimodal observations—how the events in one mode influence co-occurring events in another mode within a recent temporal context.</p> <hd id="AN0183073972-8">Current study</hd> <p>In the current study, we investigate how learners (i.e., college students majoring in math education) exhibited verbal and nonverbal contributions while collaboratively creating embodied math knowledge during the co-design activity of an online embodied learning intervention. We collected three multimodal data sources throughout the learning process to assess learners' verbal and nonverbal contributions: their verbal utterances, gestures, and machine-detected upper body movements. In order to obtain a more comprehensive, and holistic understanding of the multimodal interactions during the learning process, we conducted multimodal learning analyses through the separate application of two distinct approaches: (<reflink idref="bib1" id="ref79">1</reflink>) One is using the triangulating approach, which examines the relationships among accumulative accounts of multimodal interactions represented during the peer learning process. (<reflink idref="bib2" id="ref80">2</reflink>) The other is using the interleaving approach, which provides analytic examination of the temporal interplay between multimodal interactions occurred during the peer learning process. The significance of this research lies in its contribution to understanding the dynamic interplay of verbal and nonverbal modalities during collaborative meaning-making—a critical yet underexplored area in the field of MMLA.</p> <p>Specifically, this study addresses three key gaps in the previous research: (<reflink idref="bib1" id="ref81">1</reflink>) the need for analytic approaches that systematically capture the temporal dynamics of multimodal interactions, (<reflink idref="bib2" id="ref82">2</reflink>) the lack of practical insights into leveraging MMLA techniques to enhance understanding of collaborative learning, and (<reflink idref="bib3" id="ref83">3</reflink>) the potential to explore novel methodologies for analyzing multimodal data in TEL contexts. This study opens new horizons for future research by demonstrating the strengths and limitations of triangulating and interleaving approaches in analyzing multimodal interactions. Future studies can build upon this work by exploring its applicability across different modalities and diverse learning settings.</p> <hd id="AN0183073972-9">Research questions</hd> <p>In this study, we examine how body movements captured through motion detection, which have been empirically demonstrated to serve as a proxy for learners' engagement (Sung & Nathan, [<reflink idref="bib33" id="ref84">33</reflink>]), are linked to their verbal and gestural contributions during collaborative knowledge construction. Using mixed methods, the following research questions guide our investigation:</p> <p>RQ1: How does the triangulating approach reveal the relationship between learners' machine-detected body movements and their verbal and nonverbal interactions in the collaborative construction of embodied math knowledge during the co-design activity?</p> <p>This inquiry adopts a synchronic (i.e., static) approach to examine the extent to which a single modal stream, automatically captured in real time during the online embodied learning intervention, provides insights into the other two modal streams involved in collaborative meaning making—gesture and speech. By using the triangulating approach, we explore static relationships among these variables, identifying how one modality complements or corroborates the other. This research question generates two specific hypotheses:</p> <hd id="AN0183073972-10">H1a</hd> <p>Learner groups with larger variances of upper body movements will produce a higher number of gestures during the co-design activity compared to those with smaller variances.</p> <hd id="AN0183073972-11">H1b</hd> <p>Learner groups with larger variances of upper body movements will produce a higher number of verbal utterances they made during the co-design activity compared to those with smaller variances.</p> <p>RQ2: How does the interleaving approach reveal the multimodal discourse patterns reflected in learners' machine-detected body movements in the collaborative construction of embodied math knowledge during the co-design activity?</p> <p>Using the interleaving approach, this inquiry aims to capture how meaning-making unfolds over time through the dynamic interplay between modalities in a diachronic manner. This leads to the following hypothesis:</p> <hd id="AN0183073972-12">H2</hd> <p>Learner groups with larger variances of upper body movements will exhibit distinct multimodal discourse patterns produced during the co-design activity compared to those with smaller variances.</p> <hd id="AN0183073972-13">Methods</hd> <p>While we used the same empirical dataset as in our prior study (Sung & Nathan, [<reflink idref="bib33" id="ref85">33</reflink>]), we broadened the analytic scope to investigate the peer learning process further and implemented the framework of collaborative gestures (Walkington et al., [<reflink idref="bib38" id="ref86">38</reflink>]), as detailed in the preceding section (see 2.3. Automatic Detection of Body Movements).</p> <hd id="AN0183073972-14">Participants</hd> <p>We recruited 33 undergraduate students majoring in math education from multiple universities across the United States. Initially, participants were grouped into teams of four, but the final group size varied from two to five due to scheduling constraints, resulting in a total of 9 groups: one with two members, two with three members, five with four members, and one with five members. Participants were incentivized with either extra course credits (e.g., 1 percentage point of course credits) or monetary compensation (a $100 e-gift card) upon completing all study procedures. For those opting for extra credit, a certificate of participation was provided if approved by their instructors. The study was conducted entirely online via Zoom, with all activities being video-recorded. Our analyses included data from 33 participants, aged between 22 and 36 years. The majority of participants were female (N = 32), with a significant proportion identifying as White/Caucasian (N = 22). Additionally, most participants reported having no prior teaching experience (N = 19). Table 1 provides a detailed breakdown of the participants' demographic characteristics, including gender, self-identified race/ethnicity, age distribution, and teaching experience.</p> <p>Table 1 Demographic characteristics of participants (N = 33)</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Variable</p></th><th align="left"><p>Category</p></th><th align="left"><p>Frequency (N)</p></th><th align="left"><p>Percent (%)</p></th></tr></thead><tbody><tr><td align="left"><p>Gender</p></td><td align="left"><p>Male</p></td><td align="left"><p>1</p></td><td char="." align="char"><p>3.03</p></td></tr><tr><td align="left" /><td align="left"><p>Female</p></td><td align="left"><p>32</p></td><td char="." align="char"><p>96.97</p></td></tr><tr><td align="left"><p>Self-identified race/ethnicity</p></td><td align="left"><p>White/Caucasian</p></td><td align="left"><p>22</p></td><td char="." align="char"><p>66.67</p></td></tr><tr><td align="left" /><td align="left"><p>Hispanic</p></td><td align="left"><p>7</p></td><td char="." align="char"><p>21.21</p></td></tr><tr><td align="left" /><td align="left"><p>Asian</p></td><td align="left"><p>2</p></td><td char="." align="char"><p>6.06</p></td></tr><tr><td align="left" /><td align="left"><p>African American</p></td><td align="left"><p>1</p></td><td char="." align="char"><p>3.03</p></td></tr><tr><td align="left" /><td align="left"><p>Other</p></td><td align="left"><p>1</p></td><td char="." align="char"><p>3.03</p></td></tr><tr><td align="left"><p>Age</p></td><td align="left"><p>22</p></td><td align="left"><p>21</p></td><td char="." align="char"><p>43.75</p></td></tr><tr><td align="left" /><td align="left"><p>23</p></td><td align="left"><p>24</p></td><td char="." align="char"><p>50.00</p></td></tr><tr><td align="left" /><td align="left"><p>24</p></td><td align="left"><p>2</p></td><td char="." align="char"><p>4.17</p></td></tr><tr><td align="left" /><td align="left"><p>25</p></td><td align="left"><p>1</p></td><td char="." align="char"><p>2.08</p></td></tr><tr><td align="left" /><td align="left"><p>26–36</p></td><td align="left"><p>9</p></td><td char="." align="char"><p>18.75</p></td></tr><tr><td align="left"><p>Teaching experience</p></td><td align="left"><p>None</p></td><td align="left"><p>19</p></td><td char="." align="char"><p>57.58</p></td></tr><tr><td align="left" /><td align="left"><p>Less than 1 year</p></td><td align="left"><p>7</p></td><td char="." align="char"><p>21.21</p></td></tr><tr><td align="left" /><td align="left"><p>1–5 years</p></td><td align="left"><p>4</p></td><td char="." align="char"><p>12.12</p></td></tr><tr><td align="left" /><td align="left"><p>6–10 years</p></td><td align="left"><p>2</p></td><td char="." align="char"><p>6.06</p></td></tr><tr><td align="left" /><td align="left"><p>More than 10 years</p></td><td align="left"><p>1</p></td><td char="." align="char"><p>3.03</p></td></tr></tbody></table> </ephtml> </p> <hd id="AN0183073972-15">Materials</hd> <p></p> <hd id="AN0183073972-16">The hidden village (THV) and conjecture editor (THV-CE)</hd> <p> <emph>The Hidden Village</emph> (THV) is an interactive, 3D motion-capture simulation program designed to provide an augmented embodied geometry curriculum with gamified elements. Its goal is to facilitate embodied ways of mathematical reasoning and geometry proof production by replicating the in-game avatar's cognitively relevant body movements, called <emph>directed actions</emph>. THV Conjecture Editor (THV-CE) is integrated into THV, serving as a built-in design tool for creating new directed actions. In this study, THV-CE played a central role in the co-design activity where participants collaboratively developed mathematically relevant directed actions for given geometric conjectures. Participants used the editable avatar's upper body movements (arms and hands) to design directed actions, fostering embodied mathematical reasoning. Due to the virtual setting of the co-design activity, group members communicated their ideas verbally and gesturally within the Zoom platform (see Fig. 2), while a researcher operated THV-CE on behalf of the participants' requests.</p> <p>Graph: Fig. 2 Participants discussing to create new directed actions during the co-design activity</p> <hd id="AN0183073972-17">Procedures</hd> <p>During the intervention, participants took part in a 3.5-h-long online session, which comprised a series of activities, including: (<reflink idref="bib1" id="ref87">1</reflink>) pre-intervention individual interviews and surveys, (<reflink idref="bib2" id="ref88">2</reflink>) experiencing online gameplay of THV, (<reflink idref="bib3" id="ref89">3</reflink>) engaging in co-design activity conducted in groups using THV-CE, and (<reflink idref="bib4" id="ref90">4</reflink>) post-intervention individual interviews and surveys. While all participants completed the entire set of activities, the current investigation specifically focuses on analyzing multimodal discourse data from the co-design activity.</p> <p>This study strictly adhered to ethical guidelines, with participants providing informed consent before engaging in any research activities. All participants provided informed consent before participating in the study, which was approved by the institutional review board (IRB), and their identities have been kept confidential. Withdrawal rights were communicated, and data were securely stored, accessible only to the research team.</p> <hd id="AN0183073972-18">Data analysis</hd> <p></p> <hd id="AN0183073972-19">Identifying learning groups with larger and smaller variances in upper body movements</hd> <p>During the co-design activity, we collected learners' upper body movements, including both gestural and non-gestural motions, using <emph>PoseNet</emph>, a web-based motion-sensing tool equipped with advanced computer vision systems (Hassan et al., [<reflink idref="bib11" id="ref91">11</reflink>]). PoseNet allows for automated tracking of changes in body joint positions at a high rate (30 frames per second). To estimate the dimensions of a bounding box enclosing the upper body, we calculated its width and height by measuring the distance between the maximum and minimum x- and y-coordinate values of six body joint positions (e.g., left and right shoulders, elbows, and wrists), and then multiplied these values (see Fig. 3). We measured the variance of each learner's upper body movements separately for each of the three given conjectures during the co-design activity, resulting in a total of 99 observations (variances) corresponding to the 33 participants.</p> <p>Graph: Fig. 3 Calculation of the bounding box dimensions enclosing the upper body using the body joint positions</p> <p>In this study, a learner's upper body movement variance was determined by the variance of their bounding box dimensions. To explore the relationship between learners' upper body movements and their multimodal interactions during the co-design activity, we divided them into two groups based on the median value of variances in bounding box dimensions: (a) those with larger variances in upper body movements (henceforth, larger variances group) and (b) those with smaller variances (henceforth, smaller variances group).</p> <hd id="AN0183073972-20">Representational gestures</hd> <p>In this study, gesture coding involved several steps. Initially, we identified whether a gesture occurred and categorized it as either iconic (shape-resembling) or metaphoric (concept-resembling), both falling under the representational gesture category (Alibali & Nathan, [<reflink idref="bib2" id="ref92">2</reflink>]). We then distinguished whether a representational gesture accompanied speech or thought. If a learner produced a representational gesture while speaking, it was coded as a co-speech gesture, whereas a representational gesture produced while thinking without accompanying speech, it was coded as a co-thought gesture.</p> <p>Subsequently, we determined whether the representational gesture was a collaborative gesture, one which builds upon the other learner's reference gesture (Walkington et al., [<reflink idref="bib38" id="ref93">38</reflink>]), or whether it was an initial gesture. According to Walkington et al. ([<reflink idref="bib38" id="ref94">38</reflink>]) categorization of collaborative gestures, a learner's representational gesture was classified as collaborative gesture if it fit into any of the categories (i.e., echoing, mirroring, alternating, and joint). The present study, however, took place in an online platform via Zoom, with learners not physically co-located, the joint category was not applicable. If a learner's representational gesture was original and had no connection with the previously shown gesture by others, it was coded as an initiative gesture. Combining these features of representational gestures—whether they were co-speech or co-thought gestures and whether they were collaborative or initial gesture—I labeled them into four gestural codes: co-speech Initial, co-speech collaborative, co-thought initial, and co-thought collaborative gestures (described in more detail below).</p> <hd id="AN0183073972-21">Coding multimodal discourse during co-design activity</hd> <p>To investigate how learners with different upper body movements exhibit distinct discourse patterns while verbally and nonverbally contributing to the collaborative creation of embodied math knowledge (RQ2 using the interleaving approach), we transcribed learners' speech and gestures during the co-design activity. Since the conventional linear formats of transcripts were not optimal for representing multimodal behaviors (Evans et al., [<reflink idref="bib10" id="ref95">10</reflink>]), the alternative we chose was to transcribe learners' speech and gestures into an event-based data log with the timespan derived from the timestamp information.</p> <p>First, the initial raw transcript supplied by Zoom contained timespan information for each line, indicating when each verbal utterance started and ended. Second, we annotated learners' four types of representational gestures by observing the video recordings in order to document their temporal association with instances of speech (e.g., before, during, or after the speech). For instance, shown in Fig. 4, if a learner produced a co-speech collaborative gesture during the speech, the annotation of the gestural event was inserted within the line of the verbal utterance. If a learner produced a co-thought collaborative gesture either before or after the speech, this gestural event created a new line, which was subsequently incorporated into the transcript based on the starting timestamp. That is, the completed multimodal transcripts of co-design activity no longer have a linear time structure, but instead have a "temporally entangled" and overlapped non-linear structure, which represents the actual multimodal interactions that occurred during the collaborative learning process.</p> <p>Graph: Fig. 4 Example of the completed multimodal transcripts of co-design activity</p> <p>The multimodal transcripts of co-design activity were segmented by an event of verbal utterance, gesture, or a combination of both a verbal utterance and co-speech gesture, without pausing (3,623 segments). The transcripts of learners' verbal speech and gestures were coded in two different ways by each modality: (<reflink idref="bib1" id="ref96">1</reflink>) three verbal codes derived from the speech transcripts were employed using an automated process based on regular expression matching techniques (nCoder; Marquart et al., [<reflink idref="bib17" id="ref97">17</reflink>]); and (<reflink idref="bib2" id="ref98">2</reflink>) four gestural codes were coded by human coders (see Table 2).</p> <p>Table 2 Coding scheme for multimodal discourse during the co-design, including code descriptions, examples, and inter-rater reliability statistics</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Code</p></th><th align="left"><p>Description</p></th><th align="left"><p>Example</p></th><th align="left"><p><italic>Kappa</italic></p><p>R1 v. R2</p></th><th align="left"><p><italic>Kappa</italic></p><p>R1 v. nCoder</p></th><th align="left"><p><italic>Kappa</italic></p><p>R2 v. nCoder</p></th></tr></thead><tbody><tr><td align="left"><p><sc>Verbal: mathematical thinking</sc></p></td><td align="left"><p>Discussing mathematical concepts such as angles, lines, and conjectures</p></td><td align="left"><p>"I was thinking, maybe making a diagonal line with your arms."</p></td><td char="." align="char"><p>0.95**</p></td><td char="." align="char"><p>0.93*</p></td><td char="." align="char"><p>0.95**</p></td></tr><tr><td align="left"><p><sc>Verbal: design oriented</sc></p></td><td align="left"><p>Discussing how to visualize given geometric conjectures and design directed actions to help students' mathematical reasoning</p></td><td align="left"><p>"Maybe do diagonal first, just so it goes in the sequence of the conjecture statement."</p></td><td char="." align="char"><p>0.91*</p></td><td char="." align="char"><p>0.96**</p></td><td char="." align="char"><p>0.91*</p></td></tr><tr><td align="left"><p><sc>Verbal: consensus building</sc></p></td><td align="left"><p>Building consensus while collaboratively designing directed actions for given conjectures</p></td><td align="left"><p>"I think that'd be a pretty good demonstration of it."</p></td><td char="." align="char"><p>0.93*</p></td><td char="." align="char"><p>0.96**</p></td><td char="." align="char"><p>0.93*</p></td></tr><tr><td align="left"><p><sc>Gestural: co-speech initiative</sc></p></td><td align="left"><p>Producing a gesture accompanying speech, which was original and had no connection with the previously shown gestures by others</p></td><td align="left"><p>"I was thinking of making a parallelogram first [co-speech initiative gesture] and then moving on from there."</p></td><td char="." align="char"><p>0.85*</p></td><td char="." align="char"><p>–</p></td><td char="." align="char"><p>–</p></td></tr><tr><td align="left"><p><sc>Gestural: co-speech collab</sc></p></td><td align="left"><p>Producing a gesture accompanying speech, following up (i.e., echoing, mirroring, or alternating) with the previously shown gestures by others. Must be evidence that the person doing the echoing, mirroring, or alternating the original initiative gesturer</p></td><td align="left"><p>"I think that's a better starting point [co-speech collaborative gesture], starting at the 90 degree</p></td><td char="." align="char"><p>0.89*</p></td><td char="." align="char"><p>–</p></td><td char="." align="char"><p>–</p></td></tr><tr><td align="left"><p><sc>Gestural: co-thought initiative</sc></p></td><td align="left"><p>Producing a gesture accompanying thinking without any speech, which was original and had no connection with the previously shown gestures by others</p></td><td align="left"><p>[co-thought initiative gesture]</p></td><td char="." align="char"><p>0.82*</p></td><td char="." align="char"><p>–</p></td><td char="." align="char"><p>–</p></td></tr><tr><td align="left"><p><sc>Gestural: co-thought collab</sc></p></td><td align="left"><p>Producing a gesture accompanying thinking without any speech, following up (i.e., echoing, mirroring, or alternating) with the previously shown gestures by others. Must be evidence that the person doing the echoing, mirroring, or alternating the original initiative gesturer</p></td><td align="left"><p>[co-thought collaborative gesture]</p></td><td char="." align="char"><p>0.85*</p></td><td char="." align="char"><p>–</p></td><td char="." align="char"><p>–</p></td></tr></tbody></table> </ephtml> </p> <p>*indicates ρ(0.80) < 0.05; ** indicates ρ(0.80) < 0.01. Shaffer's <emph>ρ</emph> (rho) values mean that if the whole dataset were coded, we would expect to achieve a level of agreement of kappa > 0.80 with a Type I error rate of less than 5% or 1%, respectively. Quotations are verbal utterances, and square brackets [...] indicate gestures</p> <p>Inter-rater reliability for three verbal codes was obtained via comparisons between two human raters and nCoder, while the four gestural codes were validated between two human raters. Pairwise Cohen's <emph>kappa</emph> scores of both the verbal and nonverbal data set ranged between 0.82 ≤ κ ≤ 0.96, indicating significant agreement. Shaffer's <emph>rho</emph> values for each code showed acceptable inter-rater reliability, <emph>ρ</emph> < 0.05 (Shaffer, [<reflink idref="bib29" id="ref99">29</reflink>]). Transcript segments could receive multiple verbal and nonverbal codes.</p> <hd id="AN0183073972-22">Multimodal matrix and ENA model</hd> <p>To structure the multimodal data for analysis employing the interleaving approach (RQ2), we tabulated the multimodal matrix, a conceptual data representation suggested by Echeverría et al. ([<reflink idref="bib9" id="ref100">9</reflink>]) (see Fig. 1). In the current multimodal matrix, each segment contains all the information relevant to multimodal events (either verbal utterances or gestures) during the co-design activity. Based on a grounded analysis driven by the data, we set the size of a moving stanza window to 6 rows (the current row plus the 5 previous rows) within each group's multimodal discourse data during co-design activity.</p> <p>To investigate differences (if any) in multimodal discourse patterns during the co-design activity between the ENA networks of learners with larger variances of upper body movements and those with smaller variances (RQ2), we applied a two-tailed independent t-test to compare the positions of the plotted points in the projected ENA space. The corresponding ENA network graphs were also constructed.</p> <hd id="AN0183073972-23">Results</hd> <p></p> <hd id="AN0183073972-24">5.1 RQ1: How does the triangulating approach reveal the relationship between learners' machin...</hd> <p>To address this inquiry, we analyzed the quantity of verbal utterances and gestures produced by learner groups with larger (versus smaller) variances of upper body movements during the co-design activity. Learner's verbal and nonverbal interactions were quantified by the occurrences of gestural and verbal codes, representing meaningful and contextually relevant contributions. We calculated these occurrences separately for each of the three given conjectures discussed during the co-design activity, resulting in a total 99 observations from 33 participants. We conducted a one-tailed independent t-test, considering the predicted differences (Coolican, 2017)—larger variances of upper body movement would associate with more occurrences of gestures (H1a) and verbal utterances (H1b). The results of the t-test between the two learner groups are presented in Table 3.</p> <p>Table 3 T-test results of the number of representational gestures and verbal utterances between learner groups with larger (versus smaller) variances of upper body movements</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left" colspan="2" /><th align="left" colspan="2" /><th align="left" /><th align="left" /><th align="left"><p>(N = 99)</p></th></tr><tr><th align="left" rowspan="2" /><th align="left" colspan="2"><p>Larger variances of upper body movement</p></th><th align="left" colspan="2"><p>Smaller variances of upper body movement</p></th><th align="left" rowspan="2"><p><italic>t</italic></p></th><th align="left" rowspan="2"><p><italic>p</italic>-value</p></th><th align="left" rowspan="2"><p>Cohen's <italic>d</italic></p></th></tr><tr><th align="left"><p>M</p></th><th align="left"><p>SD</p></th><th align="left"><p>M</p></th><th align="left"><p>SD</p></th></tr></thead><tbody><tr><td align="left"><p>G_Total</p></td><td char="." align="left"><p>11.4</p></td><td char="." align="char"><p>9.09</p></td><td char="." align="char"><p>8.18</p></td><td char="." align="char"><p>6.72</p></td><td char="." align="char"><p>2.01</p></td><td char="." align="char"><p>0.02*</p></td><td char="." align="char"><p>0.40</p></td></tr><tr><td align="left"><p>G_CS_I</p></td><td char="." align="left"><p>1.78</p></td><td char="." align="char"><p>2.71</p></td><td char="." align="char"><p>1.33</p></td><td char="." align="char"><p>1.64</p></td><td char="." align="char"><p>1.01</p></td><td char="." align="char"><p>0.16</p></td><td char="." align="char"><p>0.20</p></td></tr><tr><td align="left"><p>G_CS_C</p></td><td align="left"><p>5.94</p></td><td char="." align="char"><p>6.05</p></td><td char="." align="char"><p>4.49</p></td><td char="." align="char"><p>4.33</p></td><td char="." align="char"><p>1.37</p></td><td char="." align="char"><p>0.09</p></td><td char="." align="char"><p>0.28</p></td></tr><tr><td align="left"><p>G_CT_I</p></td><td align="left"><p>0.74</p></td><td char="." align="char"><p>1.50</p></td><td char="." align="char"><p>0.33</p></td><td char="." align="char"><p>0.72</p></td><td char="." align="char"><p>1.76</p></td><td char="." align="char"><p>0.04*</p></td><td char="." align="char"><p>0.35</p></td></tr><tr><td align="left"><p>G_CT_C</p></td><td align="left"><p>2.94</p></td><td char="." align="char"><p>3.55</p></td><td char="." align="char"><p>2.04</p></td><td char="." align="char"><p>2.34</p></td><td char="." align="char"><p>1.49</p></td><td char="." align="char"><p>0.07</p></td><td char="." align="char"><p>0.30</p></td></tr><tr><td align="left"><p>V_Total</p></td><td align="left"><p>7.44</p></td><td char="." align="char"><p>6.53</p></td><td char="." align="char"><p>5.35</p></td><td char="." align="char"><p>3.86</p></td><td char="." align="char"><p>1.95</p></td><td char="." align="char"><p>0.03*</p></td><td char="." align="char"><p>0.40</p></td></tr><tr><td align="left"><p>V_MT</p></td><td char="." align="left"><p>5.08</p></td><td char="." align="char"><p>5.24</p></td><td char="." align="char"><p>3.53</p></td><td char="." align="char"><p>3.34</p></td><td char="." align="char"><p>1.76</p></td><td char="." align="char"><p>0.04*</p></td><td char="." align="char"><p>0.35</p></td></tr><tr><td align="left"><p>V_DO</p></td><td align="left"><p>1.94</p></td><td char="." align="char"><p>3.16</p></td><td char="." align="char"><p>1.39</p></td><td char="." align="char"><p>1.40</p></td><td char="." align="char"><p>1.13</p></td><td char="." align="char"><p>0.13</p></td><td char="." align="char"><p>0.23</p></td></tr><tr><td align="left"><p>V_CB</p></td><td char="." align="left"><p>2.28</p></td><td char="." align="char"><p>1.95</p></td><td char="." align="char"><p>1.71</p></td><td char="." align="char"><p>1.37</p></td><td char="." align="char"><p>1.67</p></td><td char="." align="char"><p>0.05*</p></td><td char="." align="char"><p>0.34</p></td></tr></tbody></table> </ephtml> </p> <p>*<emph>p</emph> <.05 G_Total = Total number of representational gestures, G_CS_I = Gestural: co-speech initiative, G_CS_C = Gestural: co-speech collab, G_CT_I = Gestural: co-speech collab, G_CT_C = Gestural: co-thought collab, V_Total = Total number of verbal utterances, V_MT = Verbal: mathematical thinking, V_DO = Verbal: design oriented, V_CB = Verbal: consensus building</p> <p>During the co-design activity, the learner group with larger variances produced significantly more gestures (M = 11.4, SD = 9.09) compared to the group with smaller variances (M = 8.18, SD = 6.72), with a medium effect size (H1a; Cohen's <emph>d</emph> = 0.40, <emph>t</emph> = 2.01, <emph>p</emph> < 0.05). Specifically, the learner group with larger variances exhibited significantly more co-thought initiative gestures (M = 0.74, SD = 1.50) than the group with smaller variances (M = 0.33, SD = 0.72), also with a medium effect size (Cohen's <emph>d</emph> = 0.35, <emph>t</emph> = 1.67, <emph>p</emph> < 0.05). These findings provide support for H1a, indicating that the learner group with larger variances in their co-design activity demonstrated a significantly higher frequency of gestures compared to the group with smaller variances.</p> <p>Moreover, the group with larger variances of upper body movements produced significantly more verbal utterances (M = 7.44, SD = 6.53) than the group with smaller variances (M = 5.35, SD = 3.86), with the medium effect size (H1b; Cohen's <emph>d</emph> = 0.40, t = 1.95, <emph>p</emph> < 0.05). Specifically, the group with larger variances in upper body movement produced significantly more verbal utterances concerning mathematical concepts (M = 5.08, SD = 5.24) and consensus building (M = 2.28, SD = 1.95) than the group with smaller variances (M = 3.34, SD = 1.76; M = 1.37, SD = 1.67, respectively). Both effect sizes were moderate (Cohen's <emph>d</emph> = 0.35 and 0.34, respectively). These findings provide support for H1b, showing that learner groups with larger variances in upper body movements produced a significantly higher number of verbal utterances during the co-design activity compared to those with smaller variances. This highlights the relationship between greater variability in physical engagement and increased verbal contributions within collaborative learning contexts.</p> <hd id="AN0183073972-25">5.2 RQ2: How does the interleaving approach reveal the multimodal discourse patterns reflecte...</hd> <p>For this inquiry, using mixed methods, we first conducted quantitative analyses using ENA to identify (if any) distinct discourse patterns between the learner groups (H2), and then we complemented these quantitative findings with examples from a qualitative analysis to offer further insights and validate the results.</p> <hd id="AN0183073972-26">Quantitative results using ENA models</hd> <p>We constructed ENA models using the multimodal matrix derived from the coded multimodal transcripts of learners' co-design activity. First, an ENA scatter plot (Fig. 5) was used to assess whether statistically significant differences existed in discourse patterns between the learner groups with larger (blue) and smaller (red) variances of upper body movements. Each round plotted point on the scatter plot represents an individual learner's network location of their multimodal discourse for each conjecture during the co-design activity, based on the weighted average of the unit's node weights in each network. To explore the difference between the two learner groups, we compared each group's group means, depicted by the larger square points (1 blue, 1 red) by aligning them, along the x-axis. These group means are presented with 95% confidence intervals (t-distribution) using the dashed boxes in blue and red, respectively. Non-overlapping intervals indicate statistically significant differences between the two means at the 5% level. The statistical analysis revealed significantly different multimodal discourse patterns in learners' verbal and nonverbal contributions during the co-design activity between the group with larger variances of upper body movements and the group with smaller variances, with a large effect size (H2; X̄<subs>Larger</subs> = -0.07, X̄<subs>Smaller</subs> = 0.07, <emph>t</emph>(90.56) = -3.27, p < 0.01, Cohen's <emph>d</emph> = 0.66). These findings provide strong empirical support for H2, demonstrating that larger variances in upper body movements are associated with distinct multimodal discourse patterns during collaborative activities.</p> <p>Graph: Fig. 5 ENA scatter plot: the learner group with larger (blue) and smaller (red) variances of upper body movements</p> <p>To examine which connections between verbal and nonverbal contributions accounted for the differences between two learner groups, we constructed mean epistemic networks for each group (see Fig. 6). These ENA networks visualize the network graphs of each group, with nodes representing codes listed in Table 2 and edge thicknesses indicating the relative frequencies of code co-occurrences.</p> <p>Graph: Fig. 6 Mean epistemic network graphs showing the connections made by the learner group with larger (blue lines of Panel A) and smaller (red lines of Panel C) variances and mean subtracted network (Panel B)</p> <p>As depicted in Panels A and C of Fig. 6, both the learner groups with larger (blue network) and smaller (red network) variances of upper body movements exhibited dense networks of connections, indicating active verbal and nonverbal contributions to the co-design activity in both groups. However, by subtracting one network from the other, which highlights differences in discourse patterns between the two groups, distinct patterns emerged. In the mean subtracted network (Panel B of Fig. 6), the learner group with larger variances of upper body movement demonstrated stronger connections from co-thought initiative gesture to co-speech initiative and co-speech collaborative gesture, as well as to mathematical thinking. Conversely, the group with smaller variances of upper body movement exhibited stronger connections between co-speech collaborative gesture and consensus building, as well as between co-speech initiative gesture and mathematical thinking.</p> <p>These findings suggest that learners with larger variances of upper body movements tended to establish more links from co-thought initiative gesture to other representational gestures and verbal utterances during the co-design activity, compared to those with smaller variances. On the other side, learners with smaller variances of upper body movement were more inclined to establish links from co-speech initiative gesture and co-speech collaborative gesture to other verbal utterances. Notably, the collaborative co-design process involved both learner groups in the majority of the activity (81.5%), and these results remained consistent across mixed and homogenous groupings.</p> <hd id="AN0183073972-27">Qualitative results</hd> <p>To gain a deeper understanding of the nuanced differences in meaning-making process between the learner groups with larger and smaller variances of upper body movements, we conducted qualitative analyses of learners' verbal and nonverbal interactions during the co-design activity. This analysis focused on segment-level examination of the coded multimodal transcripts.</p> <p>Figure 7 presents a vignette featuring a learner (user 17 in Group 6) with a larger variance of upper body movement, illustrating their contributions to the co-design process. In this vignette, the learners in Group 6 are collaboratively designing new directed actions for a given conjecture, stating "Consecutive angles (i.e., angles beside each other) in a parallelogram are supplementary (i.e., add up to 180 degrees)". In line 1 of the excerpt in Fig. 7, user 17 initiates the interaction with a co-thought initiative gesture, representing a sliding action with a slanted arm (Panel A; co-thought initiative code). This gesture introduces a novel idea distinct from the preceding discussion among group members about visual representations of a parallelogram. While user 17 proposes this new embodied idea, a researcher facilitates the co-design process on how to depict a parallelogram and summarizes the design-related considerations of the discussed options (lines 2–3), visually displaying the shared embodied ideas (Panel B). Subsequently, user 17 incorporates mathematical information to the co-speech collaborative gestures, generating a sequence of co-speech collaborative gestures that build upon the shared embodied ideas (co-speech collab and math thinking codes; line 4). This set of directed actions (Panel C) proposed by user 17 address the identified constraints of the discussed options and effectively convey the targeted mathematical concepts by integrating multiple embodied ideas.</p> <p>Graph: Fig. 7 Vignette 1: Example of learners' verbal and nonverbal contributions during the co-design activity, showcasing a learner with larger variance of upper body movements (Panels A & C) and researcher (Panel B)</p> <p>As illustrated by this vignette, learners with larger variances of upper body movement, who produced significantly more representational gestures, particularly co-thought initiative gestures, appeared to be more adapt at expressing their embodied ideas through co-thought gestures and fluent in integrating shared embodied ideas into their own ideation process compared to those with smaller variances.</p> <p>On the other hand, learners with smaller variances of upper body movement exhibited distinct multimodal discourse patterns in their verbal and nonverbal contributions during the co-design activity. Figure 8 presents a vignette involving two learners (users 22 and 23 in Group 4) with larger (user 22) and smaller (user 23) variances of upper body movement, as they engage in the co-design activity. In this vignette, four learners in Group 4 are collaboratively designing new directed actions for a given conjecture, stating "The diagonals of a rectangle always have the same length." User 22, with a larger variance of upper body movement, swiftly produces a co-thought initiative gesture representing a right angle on each side with two arms (Panel A). User 23, with a smaller variance, observes user 22's gesture and validates the intended mathematical concept by both replicating the gesture and verbalizing the concept, stating "I think I saw (user 22) build [co-speech collab] right angle, right? With your two arms." (co-speech collab and math thinking codes; line 3). Responding to user 23's question, user 22 confirms the intended mathematical concept and proposes it as directed actions for visualizing the given geometric conjecture by recreating the previous gesture ("[co-speech collab] Maybe we could have them (students) do that, a rectangle"; co-speech collab and math thinking codes; line 4). User 23 then agrees with user 22's proposal (consensus building code) and suggests a sequence of directed actions through co-speech collaborative gestures (co-speech collab code; Panel B) based on their shared mathematical and design-related ideas (math thinking and design oriented codes; line 5).</p> <p>Graph: Fig. 8 Vignette 2: Example of learners' verbal and nonverbal contributions during the co-design activity, showcasing a learner with larger variance of upper body movements (Panel A) and another with smaller variance (Panel B)</p> <p>As exemplified in this vignette, learners with smaller variances of upper body movements, who produced significantly fewer representational gestures, especially fewer co-thought initiative gestures, appeared to abstain from expressing their partially-formed embodied ideas during their cognitive processes. Instead, these learners predominantly focused on their peers' embodied ideas and actively contributed to advancing the design of directed actions. These qualitative findings align with the aforementioned quantitative findings using ENA models.</p> <hd id="AN0183073972-28">Discussion</hd> <p>In this paper, we discussed two distinct approaches for analyzing multimodal interactions within various TEL environments—triangulating and interleaving. By applying both approaches into an empirical dataset of peer learning processes, we exemplified how the interleaving approach can unveil aspects of collaborative meaning-making that are not readily obtained through the triangulating approach.</p> <p>Research question 1 (RQ1) asked: <emph>How does the triangulating approach reveal the relationship between learners' machine-detected body movements and their verbal and nonverbal interactions in the collaborative construction of embodied math knowledge during the co-design activity?</emph> The analysis using the triangulating approach revealed significant differences in the number of representational gestures (H1a) and verbal utterances (H1b) between groups with larger and smaller variances of upper body movements. Specifically, the group with larger variances exhibited more representational gestures, particularly co-thought initiative gestures, and more verbal utterances, especially those related to mathematical concepts and consensus building. These results provide empirical support for H1a and H1b, confirming that learners with larger variances in upper body movements tend to produce more gestures and verbal utterances during collaboration. This phenomenon may be attributed to their engagement in producing both small- and large-scale gestures, driven by a shared underlying mechanism for co-thought and co-speech gestures (Chu & Kita, [<reflink idref="bib6" id="ref101">6</reflink>], [<reflink idref="bib7" id="ref102">7</reflink>]; Kita et al., [<reflink idref="bib14" id="ref103">14</reflink>]). Consequently, automated detection of upper body movements could serve as a proxy for assessing deeper engagement in embodied learning activities, consistent with our prior research findings (Sung & Nathan, [<reflink idref="bib33" id="ref104">33</reflink>]). These findings directly address RQ1 by demonstrating how the triangulating approach effectively identifies and quantifies quantitative differences in verbal and nonverbal contributions, linking upper body movement variance to collaborative interactions in embodied knowledge co-construction.</p> <p>Research question 2 (RQ2) asked: <emph>How does the interleaving approach reveal the multimodal discourse patterns reflected in learners' machine-detected body movements in the collaborative construction of embodied math knowledge during the co-design activity?</emph> The analysis employing the interleaving approach provided insights into the distinct discourse patterns of verbal and nonverbal contributions between the two learner groups during the collaborative meaning-making process, which were not fully captured by the triangulating approach. The ENA models revealed that learners with larger and smaller variances of upper body movements exhibited statistically significant differences in their discourse patterns during the co-design activity (H2). These quantitative findings, supported by qualitative analyses, provide strong empirical evidence for H2, demonstrating that groups with larger variances exhibit distinct multimodal discourse patterns compared to those with smaller variances. These findings address Research Question 2 by highlighting how the interleaving approach uncovers the nuanced temporal interplay of multimodal contributions, providing a richer understanding of collaborative meaning-making that goes beyond static analyses.</p> <p>Furthermore, based on the ENA network models with multimodal codes, we can interpret and characterize the learners with different variances of upper body movement. Specifically, learners with larger variances displayed a stronger integration of co-thought initiative gestures, co-speech gestures (both initiative and collaborative), and verbal contributions related to mathematical reasoning, highlighting their active engagement in embodied meaning-making processes. That being said, learners with larger variances of upper body movement were <emph>embodied thinkers</emph> who appeared more comfortable expressing their cognitive engagement with embodied mathematical ideas through co-thought initiative gestures. These co-thought initiative gestures were "gestures for themselves" that helped them build their own mathematical ideas (Zurina & Williams, [<reflink idref="bib40" id="ref105">40</reflink>]), but also influenced the idea development of their peers who were able to observe these gestures. Conversely, learners with smaller variances tended to rely more on observing and integrating their peers' embodied contributions into verbal communication, reflecting a more collaborative and idea-building approach through integration rather than initiation. In other words, those with smaller variances were <emph>idea builders</emph> who focused more on observing and expanding upon their peers' embodied ideas than expressing their own partially-formed embodied ideas. These learners typically integrated their own embodied ideas into verbal communication. Together, these findings demonstrate how distinct discourse patterns reflect learners' varied approaches to co-construction of embodied knowledge, further validating the interleaving approach as a method for capturing the temporal dynamics of multimodal interactions in collaborative learning contexts.</p> <p>In this work, we demonstrated how the interleaving approach could reveal the nuances of the collaborative meaning-making processes that the triangulating approach could not, which supports its complementary role in multimodal learning analysis of peer learning processes in TEL environments. The interleaving approach is deserving of consideration when we seek to understand how shared meaning-making evolves over time during collaboration. As learning is an inherently cumulative and dynamic process that unfolds over time, the timing and order of meaning-making events during the learning process is a direct representation of how and what one learns (Reimann, [<reflink idref="bib25" id="ref106">25</reflink>]). This may be especially so in collaborative learning contexts in TEL environments, where knowledge acquisition and expression occur through multimodal interactions such as verbal utterances, gestures, digital tool use, and so on (e.g., Bridges et al., [<reflink idref="bib5" id="ref107">5</reflink>]). These multimodal events will contemporaneously occur and dynamically contribute to the meaning-making process. The triangulating approach is effective for comparing and correlating cumulative accounts of interactions that increase the validity of the findings by corroborating the observations across different modalities, thereby helping researchers obtain a more complete picture of the learning process (e.g., Noroozi et al., [<reflink idref="bib22" id="ref108">22</reflink>]). However, the triangulating approach is not designed to account for the temporal structure of the learning process, and therefore omits contextual information of the interactions. Including such information from an analysis may improve the modeling of the meaning-making process in ways that complement accounts of its overall cumulative change. Through empirical findings, we showcased how the interleaving approach unveils the intricacies of collaborative meaning-making processes that the triangulating approach cannot capture alone. This underscores the importance of considering both approaches in analyzing multimodal interactions occurring peer learning processes in TEL environments.</p> <p>This study has some limitations. First, while the bounding box dimensions enclosing a learner's upper body were used to measure gesture space, multiple factors—such as sociocultural backgrounds, gender, and personality—may contribute to differences in gesture use. As a result, we cannot conclusively determine whether representational gestures were generated primarily for the learners themselves or for their peers based solely on variations in bounding box areas. Additionally, while demographic characteristics such as gender are known to influence peer learning processes (Noroozi et al., [<reflink idref="bib23" id="ref109">23</reflink>]), the current study does not deeply investigate how these characteristics shape engagement in collaborative learning. This is because the primary focus of the study was to examine how the two distinct analytical approaches, triangulating and interleaving, reveal verbal and nonverbal interactions during the collaborative learning process. Future research could build upon this work by exploring how demographic factors interact with multimodal interactions to influence peer learning dynamics. Third, due to the online nature of the present embodied professional learning intervention, which was conducted through Zoom platform, certain collaborative embodied actions that are typically observed in co-located settings, such as engaging in joint gestures with others during collaborative activities, were not present. Subsequent research should be carried out in face-to-face environments, given that teachers may employ diverse methods of multimodal interactions within these proximal co-located situations. Fourth, while this study provides robust methodological insights, the current sample size (<emph>N</emph> = 33) may limit the generalizability of our findings. Expanding the sample in future research will help validate the patterns of multimodal interactions identified in this study and ensure broader applicability across diverse learner populations. Lastly, the findings of this study primarily focus on the collaborative meaning-making process without directly assessing its relationship to learning outcomes. Future research could build upon this work by investigating how multimodal interaction patterns, such as those observed in this study, influence individual and group learning gains, including conceptual changes or the development of positive attitudes toward embodied knowledge. This approach would provide a more comprehensive understanding of the interplay between collaborative processes and educational outcomes, ultimately reinforcing the practical implications of the interleaving and triangulating approaches.</p> <p>Despite these limitations, this study makes valuable contributions to our scientific understanding of the temporally entangled multimodal interactions during collaborative learning and improving the accurate modeling of peer learning processes in TEL environments.</p> <hd id="AN0183073972-29">Conclusion and implications for theory and research</hd> <p>This study contributes to the growing body of research on multimodal interactions in collaborative learning by employing two distinct analytical approaches—triangulating and interleaving— to uncover interconnectedness among machine-detected body movements, verbal and nonverbal interactions, and multimodal discourse patterns. The triangulating approach provides a static perspective, highlighting quantitative differences in learners' verbal and nonverbal contributions, whereas the interleaving approach captures the dynamic interplay of multimodal discourse patterns over time. Notably, the interleaving approach, utilizing ENA, reveals temporal dynamics and connections among multimodal events, advancing theoretical frameworks that integrate dynamic, temporal analyses into MMLA research. Collectively, these findings underscore the value of combining both approaches to achieve a more comprehensive understanding of multimodal interactions in collaborative learning contexts. The results of the set of hypotheses (H1a, H1b, and H2) also support the notion that learners with greater variances in upper body movements engage more deeply in embodied learning activities, as evidenced by their production of co-thought gestures and verbal utterances. These findings provide empirical evidence that automated detection of body movements can serve as a meaningful proxy for assessing engagement in collaborative learning, contributing to theory-building in MMLA, embodied cognition, and collaborative meaning-making.</p> <p>From a theoretical perspective, this study contributes to updating frameworks for embodied and collaborative learning by emphasizing the role of multimodal dynamics in peer learning processes. The observed differences in discourse patterns between groups with larger and smaller upper body movement variances suggest that multimodal interactions are not uniform but are shaped by learners' physical and cognitive engagement. Furthermore, the triangulating and interleaving approaches employed in this study serve as exemplary analytic methods for educators and researchers seeking a deeper understanding of collaborative learning processes by providing guidance for analyzing verbal and nonverbal contributions.</p> <p>From a practical standpoint, this study offers actionable insights for the design and implementation of collaborative learning environments. The findings demonstrate the potential of automated detection tools, such as web-based motion sensors, to assess learners' engagement and provide real-time feedback in collaborative learning environments. Automated motion detection technologies offers practical avenues for assessing and supporting learners in real time, enabling targeted interventions and supports for those who may otherwise remain disengaged. For example, real-time feedback systems could be developed to promote balanced verbal and nonverbal participation among learners, fostering more equitable and effective group dynamics. Additionally, the identification of distinct learner roles, such as "embodied thinkers" and "idea builders," offers practical guidance for tailoring instructional strategies to the specific needs of individuals and groups. Educators and instructional designers can leverage these insights to create learning activities and interventions that encourage both embodied and verbal forms of participation.</p> <p>Future research should expand on these findings by exploring how demographic factors, such as gender and sociocultural background, influence multimodal interactions and peer learning processes. Investigating these factors in diverse learning contexts could further refine theoretical underpinnings and enhance the applicability of automated analysis tools in educational practice. In summary, this study lays a foundation for future research and practical applications that leverage embodied and multimodal perspectives to deepen understanding of peer learning processes in TEL environments.</p> <hd id="AN0183073972-30">Acknowledgements</hd> <p>Not applicable.</p> <hd id="AN0183073972-31">Author contributions</hd> <p>HS played a pivotal role in the conceptualization of the study, design of the methodology, investigation, validation, visualization, original draft writing, software, data curation, formal analysis, funding acquisition, software application, review and editing, project administration, resource management, and supervision. MJN actively contributed to the conceptualization, review and editing, design of software, resource management, project administration, supervision, and played a crucial role in securing funding. Both authors engaged in research development and investigation. All authors collaborated in the revision and finalization of the manuscript.</p> <hd id="AN0183073972-32">Funding</hd> <p>The current research was not supported by any external funding sources.</p> <hd id="AN0183073972-33">Availability of data and materials</hd> <p>The completely anonymized datasets used and analyzed in the current study can be available from the corresponding author upon reasonable request. However, it's important to note that certain restrictions may apply to the availability of these data due to compliance with IRB standards, potentially rendering them not publicly accessible.</p> <hd id="AN0183073972-34">Declarations</hd> <p></p> <hd id="AN0183073972-35">Competing interests</hd> <p>The authors declare that they have no competing interests.</p> <hd id="AN0183073972-36">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0183073972-37"> <title> References </title> <blist> <bibl id="bib1" idref="ref19" type="bt">1</bibl> <bibtext> Alibali, M. W, & Nathan, M. J. (2007). Teachers' gestures as a means of scaffolding students' understanding: Evidence from an early algebra lesson. In Video research in the learning sciences (pp. 349–365). Routledge. https://doi.org/10.1111/j.1467-8535.2008.00890_7.x</bibtext> </blist> <blist> <bibl id="bib2" idref="ref20" type="bt">2</bibl> <bibtext> Alibali MW, Nathan MJ. Embodiment in mathematics teaching and learning: Evidence from learners' and teachers' gestures. Journal of the Learning Sciences. 2012; 21; 2: 247-286. 10.1080/10508406.2011.611446. 1187.11025</bibtext> </blist> <blist> <bibl id="bib3" idref="ref26" type="bt">3</bibl> <bibtext> Alibali MW, Nathan MJ, Church RB, Wolfgram MS, Kim S, Knuth EJ. Teachers' gestures and speech in mathematics lessons: Forging common ground by resolving trouble spots. ZDM - International Journal on Mathematics Education. 2013; 45; 3: 425-440. 10.1007/s11858-012-0476-0</bibtext> </blist> <blist> <bibl id="bib4" idref="ref1" type="bt">4</bibl> <bibtext> Blikstein, P. (2013). Multimodal learning analytics. Proceedings of the Third International Conference on Learning Analytics and Knowledge, 102–106.</bibtext> </blist> <blist> <bibl id="bib5" idref="ref11" type="bt">5</bibl> <bibtext> Bridges SM, Hmelo-Silver CE, Chan LK, Green JL, Saleh A. Dialogic intervisualizing in multimodal inquiry. International Journal of Computer-Supported Collaborative Learning. 2020; 15; 3: 283-318. 10.1007/s11412-020-09328-0</bibtext> </blist> <blist> <bibl id="bib6" idref="ref40" type="bt">6</bibl> <bibtext> Chu M, Kita S. The nature of gestures' beneficial role in spatial problem solving. Journal of Experimental Psychology: General. 2011; 140; 1: 102-116. 10.1037/a0021790. 1041.68655</bibtext> </blist> <blist> <bibl id="bib7" idref="ref46" type="bt">7</bibl> <bibtext> Chu M, Kita S. Co-thought and co-speech gestures are generated by the same action generation process. Journal of Experimental Psychology: Learning Memory and Cognition. 2016; 42; 2: 257-270. 10.1037/xlm0000168. 0312.46076</bibtext> </blist> <blist> <bibl id="bib8" idref="ref22" type="bt">8</bibl> <bibtext> Davidsen J, Ryberg T. "This is the size of one meter": Children's bodily-material collaboration. International Journal of Computer-Supported Collaborative Learning. 2017; 12; 1: 65-90. 10.1007/s11412-017-9248-8. 1384.86009</bibtext> </blist> <blist> <bibl id="bib9" idref="ref68" type="bt">9</bibl> <bibtext> Echeverría, V, Martinez-Maldonado, R, & Shum, S. B. (2019). Towards collaboration translucence: Giving meaning to multimodal group data. Conference on Human Factors in Computing Systems - Proceedings, 1–16. https://doi.org/10.1145/3290605.3300269</bibtext> </blist> <blist> <bibtext> Evans MA, Feenstra E, Ryon E, McNeill D. A multimodal approach to coding discourse: Collaboration, distributed cognition, and geometric reasoning. International Journal of Computer-Supported Collaborative Learning. 2011; 6; 2: 253-278. 10.1007/s11412-011-9113-0</bibtext> </blist> <blist> <bibtext> Hassan, J, Leong, J, & Schneider, B. (2021). Multimodal Data Collection Made Easy: The EZ-MMLA Toolkit. ACM International Conference on Learning Analytics, LAK '21.</bibtext> </blist> <blist> <bibtext> Hegarty M, Mayer S, Kriz S, Keehner M. The role of gestures in mental animation. Spatial Cognition and Computation. 2005; 5; 4: 333-356. 10.1207/s15427633scc0504_3. 1012.65045</bibtext> </blist> <blist> <bibtext> Huang, W, Walkington, C, & Nathan, M. J. (2023). Coordinating modalities of mathematical collaboration in shared VR environments. International Journal of Computer-Supported Collaborative Learning, 18(2), 163–201. https://doi.org/10.1007/s11412-023-09397-x</bibtext> </blist> <blist> <bibtext> Kita S, Alibali MW, Chu M. How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review. 2017; 124; 3: 245-266. 10.1037/rev0000059. 1470.05133</bibtext> </blist> <blist> <bibtext> Knight S, Wise AF, Chen B. Time for change: Why learning analytics needs temporal analysis. Journal of Learning Analytics. 2017; 4; 3: 7-17. 10.18608/jla.2017.43.2. 1516.65091</bibtext> </blist> <blist> <bibtext> Logan T, Lowrie T, Diezmann CM. Co-thought gestures: Supporting students to successfully navigate map tasks. Educational Studies in Mathematics. 2014; 87; 1: 87-102. 10.1007/s10649-014-9546-2</bibtext> </blist> <blist> <bibtext> Marquart, C. L, Swiecki, Z, Eagan, B, & Shaffer, D. W. (2018). ncodeR: Techniques for Automated Classifiers. R package version 0.1.2.https://cran.r-project.org/package=ncodeR</bibtext> </blist> <blist> <bibtext> Mercer N. The seeds of time: Why classroom dialogue needs a temporal analysis. Journal of the Learning Sciences. 2008; 17; 1: 33-59. 10.1080/10508400701793182. 1147.26013</bibtext> </blist> <blist> <bibtext> Mitri, D. Di, Schneider, J, Klemke, R, Specht, M, & Drachsler, H. (2019). Read between the lines: An annotation tool for multimodal data for learning. ACM International Conference Proceeding Series, 51–60. https://doi.org/10.1145/3303772.3303776</bibtext> </blist> <blist> <bibtext> Nathan MJ. Foundations of embodied learning: A paradigm for education. 2021; Routledge Press. 10.4324/9780429329098. 1464.62446</bibtext> </blist> <blist> <bibtext> Nathan MJShapiro L, Shannon S. Advancements of grounded and embodied mathematical reasoning: Theory, technology, and research methods. Routledge handbook of embodied cognition. 20242; Routledge New York. 1544.05119</bibtext> </blist> <blist> <bibtext> Noroozi O, Alikhani I, Järvelä S, Kirschner PA, Juuso I, Seppänen T. Multimodal data to design visual learning analytics for understanding regulation of learning. Computers in Human Behavior. 2019; 100: 298-304. 10.1016/j.chb.2018.12.019</bibtext> </blist> <blist> <bibtext> Noroozi O, Banihashem SK, Taghizadeh Kerman N, Parvaneh Akhteh Khaneh M, Babayi M, Ashrafi H, Biemans HJA. Gender differences in students' argumentative essay writing, peer review performance and uptake in online learning environments. Interactive Learning Environments. 2023; 31; 10: 6302-6316. 10.1080/10494820.2022.2034887</bibtext> </blist> <blist> <bibtext> Ochoa, X, Domínguez, F, Guamán, B, Maya, R, Falcones, G, & Castells, J. (2018). The RAP system: Automatic feedback of oral presentation skills using multimodal analysis and low-Cost sensors. ACM International Conference Proceeding Series, 360–364. https://doi.org/10.1145/3170358.3170406</bibtext> </blist> <blist> <bibtext> Reimann P. Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning. 2009; 4; 3: 239-257. 10.1007/s11412-009-9070-z. 1176.90314</bibtext> </blist> <blist> <bibtext> Schneider, B, Worsley, M, & Martinez-Maldonado, R. (2021). Gesture and Gaze: Multimodal Data in Dyadic Interactions. In International Handbook of Computer-Supported Collaborative Learning (pp. 625–641). https://doi.org/10.1007/978-3-030-65291-3_34</bibtext> </blist> <blist> <bibtext> Schneider B, Blikstein P. Unraveling students' interaction around a tangible interface using multimodal learning analytics. Journal of Educational Data Mining. 2015; 7; 3: 89-116. 0233.65046</bibtext> </blist> <blist> <bibtext> Schwartz DL, Black JB. Shuttling Between Depictive Models and Abstract Rules: Induction and Fallback. Cognitive Science. 1996; 20; 4: 457-497. 10.1207/s15516709cog2004_1. 0976.93503</bibtext> </blist> <blist> <bibtext> Shaffer DW. Quantitative ethnography. 2017; Cathcart Press. 07641063</bibtext> </blist> <blist> <bibtext> Shaffer DW, Collier W, Ruis AR. A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics. 2016; 3; 3: 9-45. 10.18608/jla.2016.33.3. 1338.14038</bibtext> </blist> <blist> <bibtext> Siebert-Evenstone, A. L, Collier, W, Ruis, A. R, & Shaffer, D. W. (2017). In Search of Conversational Grain Size: Modelling Semantic Structure Using Moving Stanza Windows. Journal of Learning Analytics, 4(3), 123–139. https://doi.org/10.18608/jla.2017.43.7</bibtext> </blist> <blist> <bibtext> Starr EL, Reilly JM, Schneider B. Toward using multi-modal learning analytics to support and measure collaboration in co-located dyads. Proceedings of International Conference of the Learning Sciences, ICLS. 2018; 1: 448-455</bibtext> </blist> <blist> <bibtext> Sung, H, & Nathan, M. J. (2024). Your body tells how you engage in collaboration: Machine-detected body movements as indicators of engagement in collaborative math knowledge building. British Journal of Educational Technology, 55(5), 1950–1973. https://doi.org/10.1111/bjet.13473</bibtext> </blist> <blist> <bibtext> Sung, H, Rau, M. A, & Van Veen, B. D. (2024). Development of an Intelligent Tutoring System That Assesses Internal Visualization Skills in Engineering Using Multimodal Triangulation. IEEE Transactions on Learning Technologies, 17, 1625–1638. https://doi.org/10.1109/TLT.2024.3396393</bibtext> </blist> <blist> <bibtext> Sung, H, Swart, M. I, & Nathan, M. J. (2022). Methods for analyzing temporally entangled multimodal data. In A. Weinberger, W. Chen, D. Hernández-Leo, & B. Chen (Eds.), Proceedings of the 15th International Conference on Computer Supported Collaborative Learning - CSCL 2022 (pp. 242–249). https://doi.org/10.22318/cscl2022.242</bibtext> </blist> <blist> <bibtext> Suthers DD. Technology affordances for intersubjective meaning making: A research agenda for CSCL. International Journal of Computer-Supported Collaborative Learning. 2006; 1; 3: 315-337. 10.1007/s11412-006-9660-y. 0980.68503</bibtext> </blist> <blist> <bibtext> Vygotsky LS. Mind in society: The development ofhigher psychological processes. 1978; Harvard University Press</bibtext> </blist> <blist> <bibtext> Walkington C, Chelule G, Woods D, Nathan MJ. Collaborative gesture as a case of extended mathematical cognition. Journal of Mathematical Behavior. 2019; 55; March. 10.1016/j.jmathb.2018.12.002100683</bibtext> </blist> <blist> <bibtext> Wertsch JV. Voices of the Mind. 1993; Harvard University Press. 10.2307/j.ctv1pncrnd. 11.0558.01</bibtext> </blist> <blist> <bibtext> Zurina H, Williams J. Gesturing for oneself. Educational Studies in Mathematics. 2011; 77; 2–3: 175-188. 10.1007/s10649-010-9294-x. 0842.90068</bibtext> </blist> </ref> <aug> <p>By Hanall Sung and Mitchell J. Nathan</p> <p>Reported by Author; Author</p> <p></p> <p>Hanall Sung Hanall Sung, Ph.D. studies multimodal approaches to knowledge construction, expression, and assessment in technology-enhanced learning and teaching. She focuses on designing and implementing learning technologies for STEM education and studying multimodal interactions, such as speech, gesture, digital traces, and more. Her analytic expertise spans multimodal learning analytics, discourse analysis, and human-AI interaction.</p> <p>Mitchell J. Nathan Mitchell J. Nathan, BSEE, Ph.D., Vilas Distinguished Achievement Professor of Learning Sciences at University of Wisconsin-Madison, studies how we think, teach, and learn, with emphasis on the role that language and embodied processes plays in mathematics, engineering, and integrated STEM learning, teaching, educational assessment, and design of interactive learning technologies</p> </aug> <nolink nlid="nl1" bibid="bib17" firstref="ref2"></nolink> <nolink nlid="nl2" bibid="bib27" firstref="ref3"></nolink> <nolink nlid="nl3" bibid="bib35" firstref="ref4"></nolink> <nolink nlid="nl4" bibid="bib21" firstref="ref5"></nolink> <nolink nlid="nl5" bibid="bib22" firstref="ref6"></nolink> <nolink nlid="nl6" bibid="bib32" firstref="ref7"></nolink> <nolink nlid="nl7" bibid="bib30" firstref="ref15"></nolink> <nolink nlid="nl8" bibid="bib36" firstref="ref16"></nolink> <nolink nlid="nl9" bibid="bib37" firstref="ref17"></nolink> <nolink nlid="nl10" bibid="bib39" firstref="ref18"></nolink> <nolink nlid="nl11" bibid="bib33" firstref="ref21"></nolink> <nolink nlid="nl12" bibid="bib34" firstref="ref23"></nolink> <nolink nlid="nl13" bibid="bib14" firstref="ref24"></nolink> <nolink nlid="nl14" bibid="bib26" firstref="ref27"></nolink> <nolink nlid="nl15" bibid="bib38" firstref="ref28"></nolink> <nolink nlid="nl16" bibid="bib12" firstref="ref37"></nolink> <nolink nlid="nl17" bibid="bib28" firstref="ref39"></nolink> <nolink nlid="nl18" bibid="bib40" firstref="ref42"></nolink> <nolink nlid="nl19" bibid="bib16" firstref="ref49"></nolink> <nolink nlid="nl20" bibid="bib24" firstref="ref51"></nolink> <nolink nlid="nl21" bibid="bib20" firstref="ref59"></nolink> <nolink nlid="nl22" bibid="bib25" firstref="ref60"></nolink> <nolink nlid="nl23" bibid="bib15" firstref="ref62"></nolink> <nolink nlid="nl24" bibid="bib18" firstref="ref63"></nolink> <nolink nlid="nl25" bibid="bib10" firstref="ref64"></nolink> <nolink nlid="nl26" bibid="bib19" firstref="ref65"></nolink> <nolink nlid="nl27" bibid="bib29" firstref="ref69"></nolink> <nolink nlid="nl28" bibid="bib31" firstref="ref78"></nolink> <nolink nlid="nl29" bibid="bib11" firstref="ref91"></nolink> <nolink nlid="nl30" bibid="bib23" firstref="ref109"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1461019
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Unraveling Temporally Entangled Multimodal Interactions: Investigating Verbal and Nonverbal Contributions to Collaborative Construction of Embodied Math Knowledge
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Hanall+Sung%22">Hanall Sung</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-9076-3762">0000-0001-9076-3762</externalLink>)<br /><searchLink fieldCode="AR" term="%22Mitchell+J%2E+Nathan%22">Mitchell J. Nathan</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Journal+of+Educational+Technology+in+Higher+Education%22"><i>International Journal of Educational Technology in Higher Education</i></searchLink>. 2025 22.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: BioMed Central, Ltd. Available from: Springer Nature. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: customerservice@springernature.com; Web site: https://www.springer.com/gp/biomedical-sciences
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 28
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Technology+Uses+in+Education%22">Technology Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Epistemology%22">Epistemology</searchLink><br /><searchLink fieldCode="DE" term="%22Research+and+Development%22">Research and Development</searchLink><br /><searchLink fieldCode="DE" term="%22Nonverbal+Learning%22">Nonverbal Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Verbal+Learning%22">Verbal Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Modalities%22">Learning Modalities</searchLink><br /><searchLink fieldCode="DE" term="%22Undergraduate+Students%22">Undergraduate Students</searchLink><br /><searchLink fieldCode="DE" term="%22Mathematics+Education%22">Mathematics Education</searchLink><br /><searchLink fieldCode="DE" term="%22Education+Majors%22">Education Majors</searchLink><br /><searchLink fieldCode="DE" term="%22Universities%22">Universities</searchLink><br /><searchLink fieldCode="DE" term="%22Cooperative+Learning%22">Cooperative Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Intermode+Differences%22">Intermode Differences</searchLink><br /><searchLink fieldCode="DE" term="%22Learning+Analytics%22">Learning Analytics</searchLink><br /><searchLink fieldCode="DE" term="%22Constructivism+%28Learning%29%22">Constructivism (Learning)</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1186/s41239-025-00504-6
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2365-9440
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: In various technology-enhanced learning (TEL) environments, knowledge co-creation progresses through multimodal interactions that integrate verbal and nonverbal modalities, such as speech and gestures. This study investigated two distinct analytical approaches for analyzing multimodal interactions--triangulating and interleaving--by applying them to collaborative learning processes during an online embodied mathematics intervention. The findings demonstrate that the interleaving approach captures the temporal dynamics and nuanced interplay between multimodal events, providing deeper insights into how shared meaning-making evolves over time. In contrast, the triangulating approach effectively identifies cumulative interaction patterns but does not account for their temporal structure. Specifically, the interleaving approach, employing epistemic network analysis, revealed statistically significant differences in discourse patterns between learners with larger and smaller variances in upper body movements during the co-design activity. These findings underscore the complementary value of the interleaving approach in analyzing multimodal interactions and offer practical implications for advancing understanding of collaborative learning processes in TEL environments.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1461019
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1461019
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1186/s41239-025-00504-6
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 28
    Subjects:
      – SubjectFull: Technology Uses in Education
        Type: general
      – SubjectFull: Epistemology
        Type: general
      – SubjectFull: Research and Development
        Type: general
      – SubjectFull: Nonverbal Learning
        Type: general
      – SubjectFull: Verbal Learning
        Type: general
      – SubjectFull: Learning Modalities
        Type: general
      – SubjectFull: Undergraduate Students
        Type: general
      – SubjectFull: Mathematics Education
        Type: general
      – SubjectFull: Education Majors
        Type: general
      – SubjectFull: Universities
        Type: general
      – SubjectFull: Cooperative Learning
        Type: general
      – SubjectFull: Intermode Differences
        Type: general
      – SubjectFull: Learning Analytics
        Type: general
      – SubjectFull: Constructivism (Learning)
        Type: general
    Titles:
      – TitleFull: Unraveling Temporally Entangled Multimodal Interactions: Investigating Verbal and Nonverbal Contributions to Collaborative Construction of Embodied Math Knowledge
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Hanall Sung
      – PersonEntity:
          Name:
            NameFull: Mitchell J. Nathan
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-electronic
              Value: 2365-9440
          Numbering:
            – Type: volume
              Value: 22
          Titles:
            – TitleFull: International Journal of Educational Technology in Higher Education
              Type: main
ResultId 1