View in EDS HTML Full Text PDF Full Text

Understanding the Relationship between Colleges Students' Artificial Intelligence Literacy and Higher Order Thinking Skills Using the 3P Model: The Mediating Roles of Behavioral Engagement and Peer Interaction

Saved in:
Bibliographic Details
Title: Understanding the Relationship between Colleges Students' Artificial Intelligence Literacy and Higher Order Thinking Skills Using the 3P Model: The Mediating Roles of Behavioral Engagement and Peer Interaction
Language: English
Authors: Kaili Lu, Jianrong Zhu, Feng Pang, Rustam Shadiev (ORCID 0000-0001-5571-1158)
Source: Educational Technology Research and Development. 2025 73(2):693-716.
Availability: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Peer Reviewed: Y
Page Count: 24
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Tests/Questionnaires
Education Level: Higher Education
Postsecondary Education
Descriptors: College Students, Artificial Intelligence, Digital Literacy, Thinking Skills, Models, Peer Relationship, Cooperative Learning, Interaction, Ethics, Learner Engagement, Curriculum Design
DOI: 10.1007/s11423-024-10434-1
ISSN: 1042-1629
1556-6501
Abstract: Artificial Intelligence (AI) has brought about significant changes in our lives, making AI literacy a crucial endeavor for the future. Despite its growing importance in academia, there is limited empirical research on its impact on college students' higher order thinking skills (HOTS). The present study systematically and comprehensively explores the relationship between college students' AI literacy and HOTS using the 3P (Presage-process-product) model. In this model, students' AI literacy represents the presage factors, while behavioral engagement and peer interaction serves as the process factors, and HOTS is the product factor. We gathered data from a survey of 260 college students. We utilized structural equation modeling to analyze the relationships between the 3P factors. The results showed that both AI usage and AI evaluation directly influenced HOTS and also indirectly affected HOTS through the mediating role of behavioral engagement and peer interaction. Conversely, AI awareness and AI ethics showed no direct influence on HOTS, although AI awareness impacted HOTS via peer interaction mediation. The results of the study have several theoretical and practical implications. From a theoretical perspective, this study incorporates AI literacy, behavioral engagement, peer interaction, and HOTS within the 3P model framework, shedding light on their interrelations. On a practical note, the results emphasize the need to consider AI literacy, behavioral engagement, peer interaction when designing courses to enhance HOTS in the era of AI.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1470802
Database: ERIC
Full text is not displayed to guests.
FullText Links:
  – Type: pdflink
    Url: https://content.ebscohost.com/cds/retrieve?content=AQICAHj0k_4E0hTGH8RJwT4gCJyBsGNe_WN95AvKlDbXJGqwxwGYK6YFLv5rQJbqTIGX_YTIAAAA4zCB4AYJKoZIhvcNAQcGoIHSMIHPAgEAMIHJBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDAExZwrqJIrglrRQEwIBEICBm5_pR9gRbUsmF1xMHlpqQkqCF1EgjrlGLWM7_MVJ_W33_z0nrClYxdG4p0AaSj79v2TXYs1oE3TsBRzOOr12JIKDy43GHmBnQeC8p-xd0d0nyCE2LfvEeLB2psofhhBKPsnwx-RNM-Gxbdk5zx5pYTc4CyNUOb-RzkKBKyXO8R3S3pXFMtxzMpUUCEuIyQMdz0MQuS1ZxDowUT86
Text:
  Availability: 1
  Value: <anid>AN0185099437;etr01apr.25;2025May14.02:47;v2.2.500</anid> <title id="AN0185099437-1">Understanding the relationship between colleges students' artificial intelligence literacy and higher order thinking skills using the 3P model: the mediating roles of behavioral engagement and peer interaction </title> <p>Artificial Intelligence (AI) has brought about significant changes in our lives, making AI literacy a crucial endeavor for the future. Despite its growing importance in academia, there is limited empirical research on its impact on college students' higher order thinking skills (HOTS). The present study systematically and comprehensively explores the relationship between college students' AI literacy and HOTS using the 3P (Presage-process–product) model. In this model, students' AI literacy represents the presage factors, while behavioral engagement and peer interaction serves as the process factors, and HOTS is the product factor. We gathered data from a survey of 260 college students. We utilized structural equation modeling to analyze the relationships between the 3P factors. The results showed that both AI usage and AI evaluation directly influenced HOTS and also indirectly affected HOTS through the mediating role of behavioral engagement and peer interaction. Conversely, AI awareness and AI ethics showed no direct influence on HOTS, although AI awareness impacted HOTS via peer interaction mediation. The results of the study have several theoretical and practical implications. From a theoretical perspective, this study incorporates AI literacy, behavioral engagement, peer interaction, and HOTS within the 3P model framework, shedding light on their interrelations. On a practical note, the results emphasize the need to consider AI literacy, behavioral engagement, peer interaction when designing courses to enhance HOTS in the era of AI.</p> <p>Keywords: Artificial intelligence literacy; Higher order thinking skills; Behavioral engagement; Peer interaction</p> <p>Copyright comment Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.</p> <hd id="AN0185099437-2">Introduction</hd> <p></p> <hd id="AN0185099437-3">Background</hd> <p>In the twenty-first century, higher order thinking skills (HOTS) have been established as fundamental for college students, equipping them to address intricate and unforeseen challenges in their future life and society (Huang et al., [<reflink idref="bib27" id="ref1">27</reflink>]; Lu et al., [<reflink idref="bib43" id="ref2">43</reflink>], [<reflink idref="bib46" id="ref3">46</reflink>]). Its importance has been emphasized by researchers, educators, policymakers and the general public across various disciplines, educational stages and countries (Jansen & Möller, [<reflink idref="bib30" id="ref4">30</reflink>]; Lu et al., [<reflink idref="bib44" id="ref5">44</reflink>]; Zain et al., [<reflink idref="bib67" id="ref6">67</reflink>]). As Lewis and Smith ([<reflink idref="bib38" id="ref7">38</reflink>], p. 136) stated, "higher order thinking occurs when a person takes new information and information stored in memory and interrelates and/or rearranges and extends this information to achieve a purpose or find possible answers in perplexing situations." From this point of view, the development process of higher order thinking skills is complex. Therefore, it is necessary to reveal the mechanism driving the development of higher order thinking skills, specifically, to understand how key factors in the learning process influence these skills.</p> <p>The advent of artificial intelligence (AI) technologies, such as AI-robots, virtual AI teaching assistants, intelligent tutoring systems, and generative AI-tools like ChatGPT (Azevedo et al., [<reflink idref="bib2" id="ref8">2</reflink>]; Delcker et al., [<reflink idref="bib13" id="ref9">13</reflink>]; Dwivedi et al., [<reflink idref="bib15" id="ref10">15</reflink>]; Wang et al., [<reflink idref="bib65" id="ref11">65</reflink>]), has profoundly transformed our lives, a change that is widely recognized and supported by research. We now find ourselves in what is called 'artificial intelligence era' (Davenport & Ronanki, [<reflink idref="bib12" id="ref12">12</reflink>]). To today's college students, navigating daily life without the assistance of these technologies would be challenging. The basic literacy required to survive and engage in the new age is becoming increasingly necessary, e.g. AI literacy. AI literacy is defined as "a set of competencies that enables individuals to critically evaluate AI technologies, communicate and collaborate effectively with AI, and use AI as a tool online, at home, and in the workplace" (Long & Magerko, [<reflink idref="bib42" id="ref13">42</reflink>], p. 2). There is no doubt that contemporary college students' AI literacy holds significant importance (Laupichler et al., [<reflink idref="bib35" id="ref14">35</reflink>]). This literacy encompasses not just the handling of artificial intelligence technology but also the realization of an ecological environment where humans and machines coexist harmoniously (Wang et al., [<reflink idref="bib64" id="ref15">64</reflink>]).</p> <p>For college students, AI can serve not only as a tool for content delivery but also as a catalyst that encourages questioning, analyze, and deep thinking about the presented information (van den Berg, G., & du Plessis, [<reflink idref="bib62" id="ref16">62</reflink>]). For example, AI systems could introduce complex problems and scenarios, requiring students to employ high order thinking skills such as analysis, evaluation and creation—beyond mere rote memorization or basic understanding, to select the most appropriate scenario from among the complex options (Delcker et al., [<reflink idref="bib13" id="ref17">13</reflink>]). In other words, AI poses challenges that compel students to leverage their higher order thinking skills for problem-solving. Therefore, integrating AI literacy with these critical skills is paramount.</p> <p>Although AI literacy is increasingly important, the empirical studies are still limited, especially about the influence of AI literacy on students' higher order thinking skills (Laupichler et al., [<reflink idref="bib35" id="ref18">35</reflink>]). Understanding the relationship between AI literacy and HOTS is pivotal for educators, researchers and policy makers to guide college students in cultivating AI literacy, which in turn improves HOTS. As such, the present study aims to investigate how AI literacy influences college students' higher order thinking skills.</p> <p>Learning is a complex process. To systematically explore the influence of various learning factors on HOTS in the era of artificial intelligence, it is necessary to analyze their impact across different stages of learning. In this learning process, behavioral engagement and peer interaction are two essential factors (Lane & Harris, [<reflink idref="bib34" id="ref19">34</reflink>]; Nguyen et al., [<reflink idref="bib51" id="ref20">51</reflink>]). They not only correlate with AI literacy but also with HOTS (Lein et al., [<reflink idref="bib37" id="ref21">37</reflink>]; Prior et al., [<reflink idref="bib54" id="ref22">54</reflink>]; Tugtekin & Koc, [<reflink idref="bib61" id="ref23">61</reflink>]). Consequently, they are selected as the mediating factors in the relationship between AI literacy and HOTS.</p> <p>Based on the above analyses, the 3P model is adopted in the present study as the theoretical framework to systematically and comprehensively explore the relationship between college students' AI literacy and higher order thinking skills.</p> <hd id="AN0185099437-4">Research purpose</hd> <p>Drawing from the analysis of prior research, the objectives of the present study are to:</p> <p></p> <ulist> <item> Review related theories and propose a structured research model, grounded in the 3P framework, to understand the relationship between AI literacy and higher order thinking skills in higher education.</item> <p></p> <item> Investigate the relationships between AI literacy and higher order thinking skills as laid out in the proposed structured model within a higher education context.</item> </ulist> <hd id="AN0185099437-5">Theoretical framework and hypotheses development</hd> <p></p> <hd id="AN0185099437-6">Theoretical framework</hd> <p>In the present study, the 3P model (Biggs, [<reflink idref="bib3" id="ref24">3</reflink>]) was used as the theoretical framework. This model was originally proposed by Dunkin and Biddle ([<reflink idref="bib14" id="ref25">14</reflink>]). The 3P model has been widely used and validated as the theoretical model in different learning context to explain the relationship between different learning factors. For example, Li et al. ([<reflink idref="bib40" id="ref26">40</reflink>]) used the model to explain the relationship between information literacy and self-directed learning skills (presage factors), academic emotions (process factors) and online learning engagement (product factors) among high school students. Li and Tsai ([<reflink idref="bib41" id="ref27">41</reflink>]) also adopted this model to explain the relationship between prior web experience and prior domain knowledge (presage factors), patterns of accessing time (process factors) and learning performance (product factors) of 12th-grade students learning in a text structure learning system context. Han ([<reflink idref="bib25" id="ref28">25</reflink>]) used this model to investigate the relationship between cognitive, non-cognitive, background factors and instructor presage factors, instructional process factors and learning outcomes (product factors) in interactive learning environments.</p> <p>The 3P model consists of three core components: presage, process and product. Within these components, factors classified into product dimension and are generally hypothesized as dependent variables. Conversely, factors classified into presage dimension are generally considered as the independent variables that influence the product factors either directly or indirectly, mediated by process factors (Lee et al., [<reflink idref="bib36" id="ref29">36</reflink>]). The presage factors usually exist prior to actual engagement in learning, these factors are relatively stable, learning-related, characteristics of the student, including prior knowledge, abilities, values, expectations concerning achievement, such as information literacy (Li et al., [<reflink idref="bib40" id="ref30">40</reflink>]). Process factors usually represent the ongoing approach to learning, including learning strategies and motivation (Li & Tsai, [<reflink idref="bib41" id="ref31">41</reflink>]). Product factors focus on 'how much' was learned (Biggs, [<reflink idref="bib3" id="ref32">3</reflink>]; Biggs et al., [<reflink idref="bib4" id="ref33">4</reflink>]). As shown in Fig. 1, there are relationships between presage factors, process factors and product factors, each component interacting with others.</p> <p>Graph: Fig. 1 The 3P model</p> <p>Based on the above analyses, the present study views AI literacy as presage factor, behavioral engagement and peer interaction as process factors, and higher order thinking skills as product factor.</p> <hd id="AN0185099437-7">AI literacy and process factors</hd> <p>The concept of AI literacy describes the competence of individuals in using artificial intelligence technology. It extends information literacy (Jones-Jang et al., [<reflink idref="bib31" id="ref34">31</reflink>]; Webber & Johnston, [<reflink idref="bib66" id="ref35">66</reflink>]) and digital literacy (Eshet, [<reflink idref="bib17" id="ref36">17</reflink>]; Reddy et al., [<reflink idref="bib55" id="ref37">55</reflink>]) into the era of artificial intelligence. Similar to information literacy and digital literacy, AI literacy does not require individuals to become specialist in the field of artificial intelligence. Instead, it necessitates that individuals "have the ability to properly identify, use and evaluate AI-related products under the premise of ethical standards" (Wang et al., [<reflink idref="bib64" id="ref38">64</reflink>], p. 1324). The widespread use of technologies, such as chatbots, virtual assistants, facial recognition, and translation apps, provides unprecedented opportunities for individuals to understand and integrate AI into their daily lives and learning experiences. These technologies enhance students' ability to personalize their educational journey, streamline learning tasks, and improve access to diverse resources. By leveraging AI, students can experience tailored support, efficient communication, and enriched learning environments, thereby making advanced technological solutions an integral part of their academic success and everyday life (Ng et al., [<reflink idref="bib48" id="ref39">48</reflink>]; Shadiev et al., [<reflink idref="bib57" id="ref40">57</reflink>]; Wang et al., [<reflink idref="bib65" id="ref41">65</reflink>]). Therefore, there is no doubt that AI literacy is essential for everyone (especially college students) to know and use AI as a tool to learn, work and live in the artificial intelligence era (Laupichler et al., [<reflink idref="bib35" id="ref42">35</reflink>]; Ng et al., [<reflink idref="bib48" id="ref43">48</reflink>]).</p> <p>Generally, AI literacy consists of four dimensions: artificial intelligence awareness, artificial intelligence usage, artificial intelligence evaluation and artificial intelligence ethics. Artificial intelligence awareness is a cognitive process that occurs before one uses a particular technology (Hallaq, [<reflink idref="bib24" id="ref44">24</reflink>]). It refers to the ability to identify and understand AI technologies, as well as the basic techniques and concepts behind them, in the process of using AI-related applications and services. It also relates to perceived abilities, confidence, and readiness in learning AI (Burgsteiner et al., [<reflink idref="bib5" id="ref45">5</reflink>]; Kandlhofer et al., [<reflink idref="bib32" id="ref46">32</reflink>]). Artificial intelligence usage refers to the ability to apply and utilize artificial intelligence techniques to skillfully complete tasks. This construct focuses on operational levels, including easy access to AI applications and tools, proficiency in the operation of AI applications and tools, and capable integration of different types of AI applications and tools. Artificial intelligence evaluation refers to the ability to select, analyze and critically evaluate AI applications and their results. This skill is closely related to students' higher order thinking skills. In the present study, evaluation requires a user to form accurate opinions regarding AI applications and product. Users who are able to evaluate an AI applications or product may need rich experience of using AI applications or products. Artificial intelligence ethics refers to the ability to be aware of the responsibilities and risks related to the usage of AI technologies (Wang et al., [<reflink idref="bib64" id="ref47">64</reflink>]). While AI technology offers convenience, it also compels deeper reflection on its intelligence and ethics implications (Gunkel, [<reflink idref="bib21" id="ref48">21</reflink>]). Therefore, a person knowledgeable about AI should possess the capability to accurately discern and address ethical concerns, ensuring the appropriate and correct use of AI technology.</p> <p>In the present study, students' behavioral engagement and peer interaction are selected as the process factors. Behavioral engagement is one of the important type of learning engagement. Learning engagement refers to the degree of attention, effort, involvement curiosity, interest, and enthusiasm shown by students in the learning process (Lu et al., [<reflink idref="bib45" id="ref49">45</reflink>]; Sun & Rueda, [<reflink idref="bib60" id="ref50">60</reflink>]). Behavioral engagement refers to students' involvement and participation in the learning activities inside and outside the classroom (e.g. asking questions, self-learning, and participating in discussions) (Fredricks et al., [<reflink idref="bib20" id="ref51">20</reflink>]). Peer interaction is "a form of cooperative learning that enhances the value of student-to-student interaction and results in different advantages of learning outcomes" (Christudason, [<reflink idref="bib11" id="ref52">11</reflink>]). It consists of two key components, respectively are communication and collaboration. Collaboration is the ability of two or more people to work together and share their perspectives and ideas in order to achieve a learning goal or complete a learning task (Hwang et al., [<reflink idref="bib28" id="ref53">28</reflink>]). Communication refers to the ability to "articulate thoughts and ideas effectively by using oral, written and nonverbal communication skills in a variety of forms and contexts" (Frazier & Reynolds, [<reflink idref="bib19" id="ref54">19</reflink>]). When it comes to the relationship between AI literacy and those process factors (behavioral engagement and peer interaction), previous related literature provided a lot of implications to this study (Prior et al., [<reflink idref="bib54" id="ref55">54</reflink>]; Tugtekin & Koc, [<reflink idref="bib61" id="ref56">61</reflink>]).</p> <p>In higher education, AI facilitates a range of innovative teaching approaches, including the realization of virtual and augmented reality learning environments and the personalization of the learning experience (Chen et al., [<reflink idref="bib8" id="ref57">8</reflink>]; von Ende et al., [<reflink idref="bib63" id="ref58">63</reflink>]). For example, within virtual reality environments, virtual experiments allow students to engage more boldly in learning activities without the fear of wasting consumables due to errors. This not only encourages active participation but also enables repeated experimentation, fostering a deeper understanding of concepts and principles, thereby enhancing engagement in learning (Huang et al., [<reflink idref="bib26" id="ref59">26</reflink>]). Furthermore, AI-driven personalized learning tailors activities to individual student needs and abilities, adjusting both content and pace according to the learner's progress. This customization can significantly facilitate engagement and improve learning outcomes (Southworth et al., [<reflink idref="bib59" id="ref60">59</reflink>]). Moreover, the widespread availability of generative AI allows for broader integration into various educational settings. These technological advances enable educators and students to engage with content in innovative ways that were not possible before, further enhancing student engagement (Nguyen et al., [<reflink idref="bib50" id="ref61">50</reflink>]). Based on these insights, we hypothesized that:</p> <hd id="AN0185099437-8">Hypothesis 1 (H1)</hd> <p>Students' AI literacy is positively related to behavioral engagement in higher education.</p> <p>Based on the above analyses, hypotheses of the relationship between each dimension of AI literacy and behavioral engagement were as following:</p> <hd id="AN0185099437-9">Hypothesis 1a (H1a)</hd> <p>Students' AI awareness is positively related to behavioral engagement in higher education.</p> <hd id="AN0185099437-10">Hypothesis 1b (H1b)</hd> <p>Students' AI usage is positively related to behavioral engagement in higher education.</p> <hd id="AN0185099437-11">Hypothesis 1c (H1c)</hd> <p>Students' AI evaluation is positively related to behavioral engagement in higher education.</p> <hd id="AN0185099437-12">Hypothesis 1d (H1d)</hd> <p>Students' AI ethics is positively related to behavioral engagement in higher education.</p> <p>AI plays a crucial role in higher education by enhancing communication and collaboration. Firstly, it facilitates the creation of virtual learning spaces, enabling students to effortlessly initiate online meetings with their peers from any location. This flexibility supports seamless collaboration and interaction. Secondly, AI's application extends to the development of virtual assistants or chatbots, which streamline the communication process between students and their instructors, as well as among classmates (Southworth et al., [<reflink idref="bib59" id="ref62">59</reflink>]). Some other AI-enhanced collaborative tools, such as Google Docs and Wikipedia, can significantly improve the efficiency of collaboration between students and instructors. Moreover, the widespread availability of generative AI can enhance educational accessibility and convenience, allowing students to interact with learning materials anytime and anywhere (Nguyen et al., [<reflink idref="bib50" id="ref63">50</reflink>]). Beyond simplifying interactions, these AI-driven tools necessitate and foster the development of students' critical thinking skills. They encourage students to articulate their knowledge clearly, apply it effectively, and collaborate using digital tools to address real-world challenges (Ng et al., [<reflink idref="bib49" id="ref64">49</reflink>]).</p> <p>Based on these insights, we hypothesized that:</p> <hd id="AN0185099437-13">Hypothesis 2 (H2)</hd> <p>Students' AI literacy is positively related to peer interaction in higher education.</p> <p>Based on the above analyses, hypotheses of the relationship between each dimension of AI literacy and peer interaction were as following:</p> <hd id="AN0185099437-14">Hypothesis 2a (H2a)</hd> <p>Students' AI awareness is positively related to peer interaction in higher education.</p> <hd id="AN0185099437-15">Hypothesis 2b (H2b)</hd> <p>Students' AI usage is positively related to peer interaction in higher education.</p> <hd id="AN0185099437-16">Hypothesis 2c (H2c)</hd> <p>Students' AI evaluation is positively related to peer interaction in higher education.</p> <hd id="AN0185099437-17">Hypothesis 2d (H2d)</hd> <p>Students' AI ethics is positively related to peer interaction in higher education.</p> <hd id="AN0185099437-18">Process factors and HOTS</hd> <p>Higher order thinking skills, a cornerstone of 21st-century competencies, are garnering significant attention nowadays (Huang et al., [<reflink idref="bib27" id="ref65">27</reflink>]). Hwang et al. ([<reflink idref="bib28" id="ref66">28</reflink>]) categorized higher these skills into three dimensions: critical thinking, problem solving, and creativity. Critical thinking is the ability to objectively analyze information, think clearly and rationally, and make rational judgments. Problem solving refers to the ability to identify problems, collect and analyze relevant information, and select and implement relevant solutions. Creativity refers to the ability to elaborate, refine, analyze and evaluate existing things, create new objects, and develop innovative ideas and methods. Regarding the relationship between process factors and higher order thinking skills, earlier studies offers valuable insights for the present study (Eseryel et al., [<reflink idref="bib16" id="ref67">16</reflink>]; Hwang et al., [<reflink idref="bib28" id="ref68">28</reflink>]).</p> <p>Referring to the relationship between behavioral engagement and HOTS, previous studies found that students' engagement had positive influence on problem representation in the game-based learning (Eseryel et al., [<reflink idref="bib16" id="ref69">16</reflink>]). Guo et al. ([<reflink idref="bib22" id="ref70">22</reflink>]) also found that students' classroom engagement was positively related to problem solving. Moreover, Lein et al. ([<reflink idref="bib37" id="ref71">37</reflink>]) suggested that engagement uniquely predicted mathematics problem-solving performance among seventh-grade students. Students' behavioral engagement encompasses classroom conduct, inquiry, participation in school-related activities, and interest in academic tasks (Nguyen et al., [<reflink idref="bib51" id="ref72">51</reflink>]). Engagement in such activities nurtures their problem solving skills, critical thinking skills and creativity. Thus, we hypothesized that:</p> <hd id="AN0185099437-19">Hypothesis 3 (H3)</hd> <p>Students' behavioral engagement is positively related to HOTS in higher education.</p> <p>Referring to the relationship between peer interaction and HOTS, previous studies found that peer interaction (including communication and collaboration) was positively related to HOTS (Lu et al., [<reflink idref="bib43" id="ref73">43</reflink>]), this finding was also verified by Hwang et al. ([<reflink idref="bib28" id="ref74">28</reflink>]). Thus, we hypothesized that:</p> <hd id="AN0185099437-20">Hypothesis 4 (H4)</hd> <p>Students' peer interaction is positively related to HOTS in higher education.</p> <hd id="AN0185099437-21">AI literacy and HOTS</hd> <p>AI systems, equipped with vast databases and analytical capabilities, offer students a more complex and dynamic learning environment than traditional settings. For example, AI can tailor learning experiences to individual students' styles and abilities. This personalization not only matches the content to each student's current level but also pushes them to expand their cognitive limits (Delcker et al., [<reflink idref="bib13" id="ref75">13</reflink>]). Additionally, given the complexity and frequent updates of AI applications, students must adapt and transfer skills across different software versions. Such scenarios demand the application of critical thinking, problem-solving, and other higher-order thinking skills, challenging students to effectively navigate and resolve issues (Ibna et al., [<reflink idref="bib29" id="ref76">29</reflink>]). Therefore, based on the foregoing analysis, we hypothesized that:</p> <hd id="AN0185099437-22">Hypothesis 5 (H5)</hd> <p>Students' AI literacy is positively related to HOTS in higher education.</p> <p>Based on the above analyses, hypotheses of the relationship between each dimension of AI literacy and HOTS were as following:</p> <hd id="AN0185099437-23">Hypothesis 5a (H5a)</hd> <p>Students' AI awareness is positively related to HOTS in higher education.</p> <hd id="AN0185099437-24">Hypothesis 5b (H5b)</hd> <p>Students' AI usage is positively related to HOTS in higher education.</p> <hd id="AN0185099437-25">Hypothesis 5c (H5c)</hd> <p>Students' AI evaluation is positively related to HOTS in higher education.</p> <hd id="AN0185099437-26">Hypothesis 5d (H5d)</hd> <p>Students' AI ethics is positively related to HOTS in higher education.</p> <p>Drawing from the above analyses, we have proposed the research model for this study. Figure 2 visually represents the hypothesized relationships among presage, process, and product factors.</p> <p>Graph: Fig. 2 The research model</p> <hd id="AN0185099437-27">Method</hd> <p></p> <hd id="AN0185099437-28">Research design</hd> <p>This study adopted a mixed methods research design to explore students' AI literacy, behavioral engagement, peer interaction and higher order thinking skills within a higher education context. We collected data via questionnaires and applied statistical methods to study the relationship among these research variables.</p> <hd id="AN0185099437-29">Participants</hd> <p>A total of 260 students from one university in the east of China were recruited. They were between 18 and 23 years old. Seven individuals' responses were incomplete, and so we removed them from data analysis. Thus, the valid response rate was 97.3%. Participants were distributed across different majors (e.g. arts or science) and academic level (e.g. freshman or junior). All participants had prior experience with artificial intelligence. Among the 253 participants, 152 were men (60.08%) and 101 were women (39.92%).</p> <p>Modern college students are, without a doubt, AI natives. Each of them possess both a smartphone and a laptop, devices that are equipped with intelligent applications. This means they interact daily with AI-enabled devises, continually cultivating their artificial intelligence literacy through every day experiences. For example, they might utilize real-time location services for navigation, rely on search systems for exploring unfamiliar topics, and employ virtual assistants, intelligent tutoring systems and AI chatbots to support their learning processes (Azevedo et al., [<reflink idref="bib2" id="ref77">2</reflink>]; Li, [<reflink idref="bib39" id="ref78">39</reflink>]).</p> <hd id="AN0185099437-30">Instruments</hd> <p>The questionnaire survey in this study consisted of two parts. The first part was used to collect demographic data of participants. The second part was used to measure students' perceptions of the key variables proposed in the research model (e.g. AI literacy, behavioral engagement, peer interaction, and higher order thinking skills). We used Cronbach's Alpha to assess the reliability of our questionnaire, as it evaluates how closely related a set of items is overall. This metric is widely applied in educational research to ensure that questionnaire items consistently measure the same underlying construct or trait. In the social sciences, values around 0.7–0.8 are generally considered acceptable, while values above 0.8 or 0.9 indicate good to excellent reliability (Fornell & Larcker, [<reflink idref="bib18" id="ref79">18</reflink>]).</p> <p> <emph>AI literacy</emph> Students' AI literacy (α = 0.847), including artificial intelligence awareness, artificial intelligence usage, artificial intelligence evaluation, and artificial intelligence ethics, each was measured by three items adapted from Wang et al. ([<reflink idref="bib64" id="ref80">64</reflink>]). This scale was initially developed for the university context and has since been widely utilized in various other higher education research contexts (Celik, [<reflink idref="bib6" id="ref81">6</reflink>]). Each item of the scale was scored on a five-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). One representative sample item to measure students' artificial intelligence awareness is "I can distinguish between smart devices and non-smart devices." One representative sample item to measure students' artificial intelligence usage is "I can skillfully use AI applications or products to help me with my daily work." One representative sample item to measure students' artificial intelligence evaluation is "I can evaluate the capabilities and limitations of an AI application or product after using it for a while." One representative sample item to measure students' artificial intelligence ethics is "I always comply with ethical principles when using AI applications or products."</p> <hd id="AN0185099437-31">Behavioral engagement</hd> <p>Students' behavioral engagement (α = 0.811) was measured by three items adapted from Skinner et al. ([<reflink idref="bib58" id="ref82">58</reflink>]). This scale was originally developed for measuring academic activities in classroom contexts and has been widely adopted in various learning contexts, including blended learning environments (Chiu et al., [<reflink idref="bib9" id="ref83">9</reflink>]). Each item was scored on a five-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). One representative sample item to measure students' behavioral engagement is "I try hard to do well in all of the learning activities."</p> <hd id="AN0185099437-32">Peer interaction and higher order thinking skills</hd> <p>Students' peer interaction (α = 0.805), including communication and collaboration, was measured by six and five items respectively. Students' higher order thinking skills (α = 0.897), including critical thinking, problem solving and creativity, was measured by three, four and three items respectively. Items of both peer interaction and higher order thinking skills scales were adapted from Hwang et al. ([<reflink idref="bib28" id="ref84">28</reflink>]). This scale was initially developed for mobile learning contexts and has since been widely utilized in various other learning contexts, including collaborative inquiry-based learning contexts and smart classroom learning environments (Lu et al., [<reflink idref="bib43" id="ref85">43</reflink>], [<reflink idref="bib46" id="ref86">46</reflink>]). Each item was scored on a five-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). One representative sample item to measure students' communication is "While I'm talking I think about how the other person feels". One representative sample item to measure students' collaboration is "I can finish my work efficiently when I conduct collaborative learning." One representative sample item to measure students' critical thinking is "I consider several alternatives to a problem before I answer." One representative sample item to measure students' problem solving is "I believe I can put effort into solving problems." One representative sample item to measure students' creativity is "I like to try something new."</p> <hd id="AN0185099437-33">Data collection and analysis</hd> <p>Data were collected in the middle of the spring semester during a mid-class break of the course in 2023. Permission was granted by the university to conduct the research before the survey was conducted. Participants were told that their results in the survey would not influence their academic grades of the course and their information would only be used for educational research. Participation was both voluntarily and anonymous. Approximately 5–8 min were needed to complete the questionnaire. The online questionnaire platform (https://<ulink href="http://www.wjx.cn">www.wjx.cn</ulink>) was used to collect the data. This platform is popular in China and is widely used in academic settings such as universities. All responses were then imported into SPSS 21.0 and AMOS 21.0 for data analysis. The structural equation modeling analysis was conducted to analyze the relationships between college students' AI literacy and higher order thinking skills.</p> <hd id="AN0185099437-34">Results</hd> <p></p> <hd id="AN0185099437-35">Exploratory factor analysis</hd> <p>Both exploratory factor analysis (EFA) and confirmatory factor analyses (CFA) were employed to validate the instruments adopted in the present study. An exploratory factor analysis (EFA) with varimax rotation was conducted to investigate the construct validity of each subscale. The KMO value was 0.868, and Bartlett's test of sphericity value was 3153.172 (p < 0.001). Furthermore, the structure explained 79.27% of the total variance in the model. Table 1 indicates the factor loadings of the items. Combined, these results suggest that this solution was a reasonable interpretation of the present survey.</p> <p>Table 1 Exploratory factor analysis results (only factor loadings greater than 0.5 are shown)</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" rowspan="2" /><th align="left" colspan="7"><p>Component</p></th></tr><tr><th align="left"><p>1</p></th><th align="left"><p>2</p></th><th align="left"><p>3</p></th><th align="left"><p>4</p></th><th align="left"><p>5</p></th><th align="left"><p>6</p></th><th align="left"><p>7</p></th></tr></thead><tbody><tr><td align="left"><p>AW1</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.876</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>AW2</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.814</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>AW3</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.829</p></td><td align="left" /><td align="left" /></tr><tr><td align="left"><p>US1</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.842</p></td><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>US2</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.845</p></td><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>US3</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.832</p></td><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>EV1</p></td><td align="left" /><td align="left"><p>0.803</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>EV2</p></td><td align="left" /><td align="left"><p>0.854</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>EV3</p></td><td align="left" /><td align="left"><p>0.813</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>ET1</p></td><td align="left" /><td align="left" /><td align="left"><p>0.875</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>ET2</p></td><td align="left" /><td align="left" /><td align="left"><p>0.91</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>ET3</p></td><td align="left" /><td align="left" /><td align="left"><p>0.886</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>BE1</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.745</p></td></tr><tr><td align="left"><p>BE2</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.613</p></td></tr><tr><td align="left"><p>BE3</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.765</p></td></tr><tr><td align="left"><p>PI1</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.772</p></td><td align="left" /></tr><tr><td align="left"><p>PI2</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.793</p></td><td align="left" /></tr><tr><td align="left"><p>PI3</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left"><p>0.755</p></td><td align="left" /></tr><tr><td align="left"><p>HOTS1</p></td><td align="left"><p>0.805</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>HOTS2</p></td><td align="left"><p>0.886</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr><tr><td align="left"><p>HOTS3</p></td><td align="left"><p>0.872</p></td><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /><td align="left" /></tr></tbody></table> </ephtml> </p> <p> <emph>EV</emph> Evaluation, <emph>US</emph> Usage, <emph>ET</emph> Ethics, <emph>AW</emph> Awareness, <emph>PI</emph> Peer interaction, <emph>BE</emph> Behavioral engagement, <emph>HOTS</emph> Higher order thinking skills</p> <hd id="AN0185099437-36">Confirmatory factor analysis</hd> <p>Following the EFA, CFA was conducted to validate the instruments. In the present study, a structural equation model (SEM) was developed and tested. The measurement model was assessed by goodness-of-fit, construct reliability and construct validity. The details of CFA are presented in Tables 2, 3 and 4. Goodness-of-fit was determined based on GFI, CFI, TLI, RMSEA and SRMR. As presented in Table 2, all model fit statistics were within the acceptable ranges (x<sups>2</sups>/df = 1.487 < 3, GFI = 0.917 > 0.90, CFI = 0.973 > 0.90, TLI = 0.966 > 0.90, RMSEA = 0.044 < 0.08, SRMR = 0.044 < 0.08) (Hair et al., [<reflink idref="bib23" id="ref87">23</reflink>]), which demonstrates that the measurement model exhibits satisfactory values.</p> <p>Table 2 Results of goodness-of-fit</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>x<sup>2</sup></p></th><th align="left"><p>Df</p></th><th align="left"><p>x<sup>2</sup>/df</p></th><th align="left"><p>GFI</p></th><th align="left"><p>CFI</p></th><th align="left"><p>TLI</p></th><th align="left"><p>RMSEA</p></th><th align="left"><p>SRMR</p></th></tr></thead><tbody><tr><td align="left"><p>Model</p></td><td char="." align="char"><p>251.32</p></td><td char="." align="char"><p>169.000</p></td><td align="left"><p>1.487</p></td><td char="." align="char"><p>0.917</p></td><td char="." align="char"><p>0.973</p></td><td char="." align="char"><p>0.966</p></td><td char="." align="char"><p>0.044</p></td><td char="." align="char"><p>0.044</p></td></tr><tr><td align="left"><p>Acceptable range</p></td><td char="." align="char" /><td char="." align="char" /><td align="left"><p> < 3</p></td><td char="." align="char"><p> ≥ 0.90</p></td><td char="." align="char"><p> ≥ 0.90</p></td><td char="." align="char"><p> ≥ 0.90</p></td><td char="." align="char"><p> ≤ 0.08</p></td><td char="." align="char"><p> ≤ 0.08</p></td></tr></tbody></table> </ephtml> </p> <p>Table 3 Results of construct reliability and validity</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>Convergent validity</p></th><th align="left" colspan="2"><p>Reliability</p></th></tr><tr><th align="left" /><th align="left"><p>AVE</p></th><th align="left"><p>CR</p></th><th align="left"><p>Cronbach's alpha</p></th></tr></thead><tbody><tr><td align="left"><p>Awareness</p></td><td char="." align="char"><p>0.685</p></td><td char="." align="char"><p>0.867</p></td><td char="." align="char"><p>0.867</p></td></tr><tr><td align="left"><p>Usage</p></td><td char="." align="char"><p>0.682</p></td><td char="." align="char"><p>0.866</p></td><td char="." align="char"><p>0.867</p></td></tr><tr><td align="left"><p>Evaluation</p></td><td char="." align="char"><p>0.717</p></td><td char="." align="char"><p>0.883</p></td><td char="." align="char"><p>0.882</p></td></tr><tr><td align="left"><p>Ethics</p></td><td char="." align="char"><p>0.733</p></td><td char="." align="char"><p>0.892</p></td><td char="." align="char"><p>0.891</p></td></tr><tr><td align="left"><p>Peer interaction</p></td><td char="." align="char"><p>0.586</p></td><td char="." align="char"><p>0.809</p></td><td char="." align="char"><p>0.805</p></td></tr><tr><td align="left"><p>Behavioral engagement</p></td><td char="." align="char"><p>0.587</p></td><td char="." align="char"><p>0.810</p></td><td char="." align="char"><p>0.811</p></td></tr><tr><td align="left"><p>Higher order thinking skills</p></td><td char="." align="char"><p>0.741</p></td><td char="." align="char"><p>0.896</p></td><td char="." align="char"><p>0.897</p></td></tr><tr><td align="left"><p>Criteria</p></td><td char="." align="char"><p> > 0.50</p></td><td char="." align="char"><p> > 0.70</p></td><td char="." align="char"><p> > 0.70</p></td></tr></tbody></table> </ephtml> </p> <p>Table 4 Results of discriminant validity</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" /><th align="left"><p>EV</p></th><th align="left"><p>US</p></th><th align="left"><p>ET</p></th><th align="left"><p>AW</p></th><th align="left"><p>PI</p></th><th align="left"><p>BE</p></th><th align="left"><p>HOTS</p></th></tr></thead><tbody><tr><td align="left"><p>EV</p></td><td char="." align="char"><p>0.846</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /></tr><tr><td align="left"><p>US</p></td><td char="." align="char"><p>0.339</p></td><td char="." align="char"><p>0.826</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /></tr><tr><td align="left"><p>ET</p></td><td char="." align="char"><p>0.258</p></td><td char="." align="char"><p>0.265</p></td><td char="." align="char"><p>0.856</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /></tr><tr><td align="left"><p>AW</p></td><td char="." align="char"><p>0.506</p></td><td char="." align="char"><p>0.447</p></td><td char="." align="char"><p>0.149</p></td><td char="." align="char"><p>0.828</p></td><td char="." align="char" /><td char="." align="char" /><td char="." align="char" /></tr><tr><td align="left"><p>PI</p></td><td char="." align="char"><p>0.507</p></td><td char="." align="char"><p>0.496</p></td><td char="." align="char"><p>0.243</p></td><td char="." align="char"><p>0.468</p></td><td char="." align="char"><p>0.766</p></td><td char="." align="char" /><td char="." align="char" /></tr><tr><td align="left"><p>BE</p></td><td char="." align="char"><p>0.636</p></td><td char="." align="char"><p>0.622</p></td><td char="." align="char"><p>0.331</p></td><td char="." align="char"><p>0.532</p></td><td char="." align="char"><p>0.491</p></td><td char="." align="char"><p>0.766</p></td><td char="." align="char" /></tr><tr><td align="left"><p>HOTS</p></td><td char="." align="char"><p>0.546</p></td><td char="." align="char"><p>0.186</p></td><td char="." align="char"><p>0.225</p></td><td char="." align="char"><p>0.264</p></td><td char="." align="char"><p>0.447</p></td><td char="." align="char"><p>0.527</p></td><td char="." align="char"><p>0.861</p></td></tr></tbody></table> </ephtml> </p> <p> <emph>EV</emph> Evaluation, <emph>US</emph> Usage, <emph>ET</emph> Ethics, <emph>AW</emph> Awareness, <emph>PI</emph> Peer interaction, <emph>BE</emph> Behavioral engagement, <emph>HOTS</emph> Higher order thinking skills</p> <p>As shown in Table 3, construct reliability of the model was determined through composite reliability (CR) and Cronbach's alpha. First, the composite reliability values were all over 0.7, demonstrating satisfactory reliability (Nunnally & Bernstein, [<reflink idref="bib52" id="ref88">52</reflink>]). Second, the Cronbach's alpha value were all over 0.7 and within acceptable limits (Fornell & Larcker, [<reflink idref="bib18" id="ref89">18</reflink>]). Convergent validity was determined through average variance extracted (AVE). The results of average variance extracted (AVE) were all greater than 0.5, which was also satisfactory (Segars, [<reflink idref="bib56" id="ref90">56</reflink>]).</p> <p>As presented in Table 4, the square roots of AVE were compared to correlations among latent variables to evaluate the discriminant validity (Fornell & Larcker, [<reflink idref="bib18" id="ref91">18</reflink>]). All latent correlations were less than the corresponding AVE square roots, which was satisfactory. Overall, these results indicate that the proposed research model has a good fit.</p> <hd id="AN0185099437-37">The structural model and hypothesis test</hd> <p>The structural model was conducted to test the hypotheses. A structural model was used to present the results of the structural modeling analysis. As can be seen in Fig. 3, the structural model was adjusted based on the research model, presented with the path coefficients marked by standardized regression weights (β value) and t-values to show the relationships between artificial intelligence literacy and higher order thinking skills. In Fig. 3, the solid line between every two variables represents that the hypothesis was supported, while the dashed line represents that the hypothesis was unsupported.</p> <p>Graph: Fig. 3 The structural model</p> <p>As can be seen in Table 5, findings reflect that some hypotheses were supported, including H2a, H1b, H2b, H1c, H2c, H5c, H3 and H4. While others were not supported, including H1a, H5a, H5b, H1d, H2d and H5d. AI awareness (β = 0.156, p < 0.05), AI usage (β = 0.309, p < 0.001) and AI evaluation (β = 0.298, p < 0.001) were significantly positively related to peer interaction. AI usage (β = 0.329, p < 0.001) and AI evaluation (β = 0.324, p < 0.001) were significantly positively related to behavioral engagement. AI evaluation (β = 0.296, p < 0.01), behavioral engagement (β = 0.616, p < 0.001) and peer interaction (β = 0.311, p < 0.01) were significantly positively associated with higher order thinking skills. In addition, it is worth mentioning that AI usage (β = -0.313, p < 0.01) had significantly negative association with higher order thinking skills.</p> <p>Table 5 Results of the hypotheses</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left"><p>Hypothesis</p></th><th align="left"><p>Relationship</p></th><th align="left"><p>β</p></th><th align="left"><p>S.E</p></th><th align="left"><p>T</p></th><th align="left"><p>Result</p></th><th align="left"><p>η<sup>2</sup></p></th></tr></thead><tbody><tr><td align="left"><p>H1a</p></td><td align="left"><p>AW → BE</p></td><td char="." align="char"><p>0.097</p></td><td char="." align="char"><p>0.054</p></td><td char="." align="char"><p>1.787</p></td><td align="left"><p>Not Supported</p></td><td char="." align="char"><p>0.210</p></td></tr><tr><td align="left"><p>H2a</p></td><td align="left"><p>AW → PI</p></td><td char="." align="char"><p>0.156</p></td><td char="." align="char"><p>0.076</p></td><td char="." align="char"><p>2.055*</p></td><td align="left"><p>Supported</p></td><td char="." align="char"><p>0.156</p></td></tr><tr><td align="left"><p>H5a</p></td><td align="left"><p>AW → HOTS</p></td><td char="." align="char"><p>-0.124</p></td><td char="." align="char"><p>0.082</p></td><td char="." align="char"><p>-1.502</p></td><td align="left"><p>Not Supported</p></td><td char="." align="char"><p>0.062</p></td></tr><tr><td align="left"><p>H1b</p></td><td align="left"><p>US → BE</p></td><td char="." align="char"><p>0.329</p></td><td char="." align="char"><p>0.059</p></td><td char="." align="char"><p>5.531***</p></td><td align="left"><p>Supported</p></td><td char="." align="char"><p>0.317</p></td></tr><tr><td align="left"><p>H2b</p></td><td align="left"><p>US → PI</p></td><td char="." align="char"><p>0.309</p></td><td char="." align="char"><p>0.082</p></td><td char="." align="char"><p>3.762***</p></td><td align="left"><p>Supported</p></td><td char="." align="char"><p>0.224</p></td></tr><tr><td align="left"><p>H5b</p></td><td align="left"><p>US → HOTS</p></td><td char="." align="char"><p>-0.313</p></td><td char="." align="char"><p>0.105</p></td><td char="." align="char"><p>-2.988**</p></td><td align="left"><p>Not Supported</p></td><td char="." align="char"><p>0.104</p></td></tr><tr><td align="left"><p>H1c</p></td><td align="left"><p>EV → BE</p></td><td char="." align="char"><p>0.324</p></td><td char="." align="char"><p>0.059</p></td><td char="." align="char"><p>5.463***</p></td><td align="left"><p>Supported</p></td><td char="." align="char"><p>0.389</p></td></tr><tr><td align="left"><p>H2c</p></td><td align="left"><p>EV → PI</p></td><td char="." align="char"><p>0.298</p></td><td char="." align="char"><p>0.079</p></td><td char="." align="char"><p>3.774***</p></td><td align="left"><p>Supported</p></td><td char="." align="char"><p>0.233</p></td></tr><tr><td align="left"><p>H5c</p></td><td align="left"><p>EV → HOTS</p></td><td char="." align="char"><p>0.296</p></td><td char="." align="char"><p>0.104</p></td><td char="." align="char"><p>2.846**</p></td><td align="left"><p>Supported</p></td><td char="." align="char"><p>0.314</p></td></tr><tr><td align="left"><p>H1d</p></td><td align="left"><p>ET → BE</p></td><td char="." align="char"><p>0.078</p></td><td char="." align="char"><p>0.046</p></td><td char="." align="char"><p>1.694</p></td><td align="left"><p>Not Supported</p></td><td char="." align="char"><p>0.204</p></td></tr><tr><td align="left"><p>H2d</p></td><td align="left"><p>ET → PI</p></td><td char="." align="char"><p>0.057</p></td><td char="." align="char"><p>0.064</p></td><td char="." align="char"><p>0.89</p></td><td align="left"><p>Not Supported</p></td><td char="." align="char"><p>0.088</p></td></tr><tr><td align="left"><p>H5d</p></td><td align="left"><p>ET → HOTS</p></td><td char="." align="char"><p>0.035</p></td><td char="." align="char"><p>0.069</p></td><td char="." align="char"><p>0.502</p></td><td align="left"><p>Not Supported</p></td><td char="." align="char"><p>0.090</p></td></tr><tr><td align="left"><p>H3</p></td><td align="left"><p>BE → HOTS</p></td><td char="." align="char"><p>0.616</p></td><td char="." align="char"><p>0.174</p></td><td char="." align="char"><p>3.546***</p></td><td align="left"><p>Supported</p></td><td char="." align="char"><p>0.245</p></td></tr><tr><td align="left"><p>H4</p></td><td align="left"><p>PI → HOTS</p></td><td char="." align="char"><p>0.311</p></td><td char="." align="char"><p>0.099</p></td><td char="." align="char"><p>3.148**</p></td><td align="left"><p>Supported</p></td><td char="." align="char"><p>0.218</p></td></tr></tbody></table> </ephtml> </p> <p> <emph>EV</emph> Evaluation, <emph>US</emph> Usage, <emph>ET</emph> Ethics, <emph>AW</emph> Awareness, <emph>PI</emph> Peer interaction, <emph>BE</emph> Behavioral engagement, <emph>HOTS</emph> Higher order thinking skills *p < 0.05; **p < 0.01; ***p < 0.001</p> <hd id="AN0185099437-38">Analysis of indirect effects among key factors</hd> <p>To investigate the mediation effect, the direct and indirect effect of each hypothesis was analyzed in the present study. Based on the above analyses, we performed percentile bootstrapping and bias-corrected bootstrapping at 95% confidence interval with 2000 bootstrap sample (Arnold et al., [<reflink idref="bib1" id="ref92">1</reflink>]) to test full or partial mediation. We calculated the confidence of interval of the lower and upper bounds to test the significance of indirect effects as recommended by Preacher and Hayes ([<reflink idref="bib53" id="ref93">53</reflink>]).</p> <p>As shown in Table 6, the indirect effects of AI awareness on higher order thinking skills (β = 0.05, p < 0.05, Z = 1.66) via peer interaction, AI usage on higher order thinking skills (β = 0.20, p < 0.01, Z = 2.82) via behavioral engagement, AI usage on higher order thinking skills (β = 0.10, p < 0.01, Z = 2.59) via peer interaction, AI evaluation on higher order thinking skills (β = 0.20, p < 0.01, Z = 2.84) via behavioral engagement and AI evaluation on higher order thinking skills (β = 0.09, p < 0.01, Z = 2.66) via peer interaction were all significant. While the indirect effect of AI awareness on higher order thinking skills (β = 0.06, p = 0.09, Z = 1.40) via behavioral engagement, AI ethics on higher order thinking skills (β = 0.05, p = 0.20, Z = 1.09) via behavioral engagement and AI ethics on higher order thinking skills (β = 0.02, p = 0.35, Z = 0.86) via peer interaction were not significant.</p> <p>Table 6 Bootstrapping results for standardized direct, indirect and total effects of the hypothesized model</p> <p> <ephtml> <table frame="hsides" rules="groups"><thead><tr><th align="left" rowspan="3" /><th align="left" rowspan="3"><p>β</p></th><th align="left" rowspan="3"><p>SE</p></th><th align="left" rowspan="3"><p>Z</p></th><th align="left" colspan="5"><p>Bootstrapping</p></th></tr><tr><th align="left" colspan="2"><p>95% CI</p></th><th align="left" colspan="2"><p>Bias-corrected</p><p>95% CI</p></th><th align="left" rowspan="2"><p>Two-tailed</p><p>significance</p></th></tr><tr><th align="left"><p>LLCI</p></th><th align="left"><p>ULCI</p></th><th align="left"><p>LLCI</p></th><th align="left"><p>ULCI</p></th></tr></thead><tbody><tr><td align="left" colspan="9"><p>Standardized direct effects</p></td></tr><tr><td align="left"><p> AW → HOTS</p></td><td align="left"><p>− 0.12</p></td><td char="." align="char"><p>0.09</p></td><td align="left"><p>− 1.39</p></td><td align="left"><p>− 0.29</p></td><td char="." align="char"><p>0.05</p></td><td align="left"><p>− 0.29</p></td><td char="." align="char"><p>0.05</p></td><td char="." align="char"><p>0.16</p></td></tr><tr><td align="left"><p> ET → HOTS</p></td><td align="left"><p>0.03</p></td><td char="." align="char"><p>0.07</p></td><td align="left"><p>0.46</p></td><td align="left"><p>− 0.12</p></td><td char="." align="char"><p>0.16</p></td><td align="left"><p>− 0.11</p></td><td char="." align="char"><p>0.16</p></td><td char="." align="char"><p>0.61</p></td></tr><tr><td align="left"><p> US → HOTS</p></td><td align="left"><p>− 0.27</p></td><td char="." align="char"><p>0.10</p></td><td align="left"><p>− 2.81</p></td><td align="left"><p>− 0.48</p></td><td char="." align="char"><p>− 0.10</p></td><td align="left"><p>− 0.46</p></td><td char="." align="char"><p>− 0.08</p></td><td char="." align="char"><p>0.008(**)</p></td></tr><tr><td align="left"><p> EV → HOTS</p></td><td align="left"><p>0.27</p></td><td char="." align="char"><p>0.13</p></td><td align="left"><p>2.13</p></td><td align="left"><p>0.00</p></td><td char="." align="char"><p>0.50</p></td><td align="left"><p>0.03</p></td><td char="." align="char"><p>0.53</p></td><td char="." align="char"><p>0.029(*)</p></td></tr><tr><td align="left"><p> BE → HOTS</p></td><td align="left"><p>0.44</p></td><td char="." align="char"><p>0.14</p></td><td align="left"><p>3.14</p></td><td align="left"><p>0.20</p></td><td char="." align="char"><p>0.76</p></td><td align="left"><p>0.18</p></td><td char="." align="char"><p>0.73</p></td><td char="." align="char"><p>0.004(**)</p></td></tr><tr><td align="left"><p> PE → HOTS</p></td><td align="left"><p>0.28</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>3.34</p></td><td align="left"><p>0.12</p></td><td char="." align="char"><p>0.45</p></td><td align="left"><p>0.12</p></td><td char="." align="char"><p>0.45</p></td><td char="." align="char"><p>0.001(**)</p></td></tr><tr><td align="left"><p> AW → BE</p></td><td align="left"><p>0.13</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>1.59</p></td><td align="left"><p>− 0.05</p></td><td char="." align="char"><p>0.28</p></td><td align="left"><p>− 0.03</p></td><td char="." align="char"><p>0.29</p></td><td char="." align="char"><p>0.11</p></td></tr><tr><td align="left"><p> ET → BE</p></td><td align="left"><p>0.10</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>1.20</p></td><td align="left"><p>− 0.06</p></td><td char="." align="char"><p>0.26</p></td><td align="left"><p>− 0.07</p></td><td char="." align="char"><p>0.25</p></td><td char="." align="char"><p>0.27</p></td></tr><tr><td align="left"><p> US → BE</p></td><td align="left"><p>0.40</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>4.98</p></td><td align="left"><p>0.24</p></td><td char="." align="char"><p>0.55</p></td><td align="left"><p>0.24</p></td><td char="." align="char"><p>0.56</p></td><td char="." align="char"><p>0.000(**)</p></td></tr><tr><td align="left"><p> EV → BE</p></td><td align="left"><p>0.41</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>5.31</p></td><td align="left"><p>0.27</p></td><td char="." align="char"><p>0.57</p></td><td align="left"><p>0.25</p></td><td char="." align="char"><p>0.55</p></td><td char="." align="char"><p>0.001(**)</p></td></tr><tr><td align="left"><p> AW → PI</p></td><td align="left"><p>0.17</p></td><td char="." align="char"><p>0.09</p></td><td align="left"><p>1.98</p></td><td align="left"><p>− 0.02</p></td><td char="." align="char"><p>0.33</p></td><td align="left"><p>− 0.01</p></td><td char="." align="char"><p>0.33</p></td><td char="." align="char"><p>0.07</p></td></tr><tr><td align="left"><p> ET → PI</p></td><td align="left"><p>0.06</p></td><td char="." align="char"><p>0.07</p></td><td align="left"><p>0.86</p></td><td align="left"><p>− 0.08</p></td><td char="." align="char"><p>0.20</p></td><td align="left"><p>− 0.09</p></td><td char="." align="char"><p>0.19</p></td><td char="." align="char"><p>0.42</p></td></tr><tr><td align="left"><p> US → PI</p></td><td align="left"><p>0.30</p></td><td char="." align="char"><p>0.09</p></td><td align="left"><p>3.47</p></td><td align="left"><p>0.13</p></td><td char="." align="char"><p>0.47</p></td><td align="left"><p>0.13</p></td><td char="." align="char"><p>0.47</p></td><td char="." align="char"><p>0.002(**)</p></td></tr><tr><td align="left"><p> EV → PI</p></td><td align="left"><p>0.30</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>3.84</p></td><td align="left"><p>0.16</p></td><td char="." align="char"><p>0.47</p></td><td align="left"><p>0.14</p></td><td char="." align="char"><p>0.45</p></td><td char="." align="char"><p>0.001(**)</p></td></tr><tr><td align="left" colspan="9"><p>Standardized indirect effects</p></td></tr><tr><td align="left"><p> AW → BE → HOTS</p></td><td align="left"><p>0.06</p></td><td char="." align="char"><p>0.04</p></td><td align="left"><p>1.40</p></td><td align="left"><p>− 0.03</p></td><td char="." align="char"><p>0.15</p></td><td align="left"><p>− 0.01</p></td><td char="." align="char"><p>0.16</p></td><td char="." align="char"><p>0.09</p></td></tr><tr><td align="left"><p> AW → PI → HOTS</p></td><td align="left"><p>0.05</p></td><td char="." align="char"><p>0.03</p></td><td align="left"><p>1.66</p></td><td align="left"><p>0.00</p></td><td char="." align="char"><p>0.11</p></td><td align="left"><p>0.00</p></td><td char="." align="char"><p>0.12</p></td><td char="." align="char"><p>0.049(*)</p></td></tr><tr><td align="left"><p> US → BE → HOTS</p></td><td align="left"><p>0.20</p></td><td char="." align="char"><p>0.07</p></td><td align="left"><p>2.82</p></td><td align="left"><p>0.07</p></td><td char="." align="char"><p>0.35</p></td><td align="left"><p>0.07</p></td><td char="." align="char"><p>0.35</p></td><td char="." align="char"><p>0.001(**)</p></td></tr><tr><td align="left"><p> US → PI → HOTS</p></td><td align="left"><p>0.10</p></td><td char="." align="char"><p>0.04</p></td><td align="left"><p>2.59</p></td><td align="left"><p>0.03</p></td><td char="." align="char"><p>0.17</p></td><td align="left"><p>0.03</p></td><td char="." align="char"><p>0.19</p></td><td char="." align="char"><p>0.001(**)</p></td></tr><tr><td align="left"><p> EV → BE → HOTS</p></td><td align="left"><p>0.20</p></td><td char="." align="char"><p>0.07</p></td><td align="left"><p>2.84</p></td><td align="left"><p>0.08</p></td><td char="." align="char"><p>0.35</p></td><td align="left"><p>0.07</p></td><td char="." align="char"><p>0.34</p></td><td char="." align="char"><p>0.002(**)</p></td></tr><tr><td align="left"><p> EV → PI → HOTS</p></td><td align="left"><p>0.09</p></td><td char="." align="char"><p>0.04</p></td><td align="left"><p>2.66</p></td><td align="left"><p>0.03</p></td><td char="." align="char"><p>0.17</p></td><td align="left"><p>0.03</p></td><td char="." align="char"><p>0.17</p></td><td char="." align="char"><p>0.001(**)</p></td></tr><tr><td align="left"><p> ET → BE → HOTS</p></td><td align="left"><p>0.05</p></td><td char="." align="char"><p>0.04</p></td><td align="left"><p>1.09</p></td><td align="left"><p>− 0.03</p></td><td char="." align="char"><p>0.15</p></td><td align="left"><p>− 0.03</p></td><td char="." align="char"><p>0.15</p></td><td char="." align="char"><p>0.20</p></td></tr><tr><td align="left"><p> ET → PI → HOTS</p></td><td align="left"><p>0.02</p></td><td char="." align="char"><p>0.02</p></td><td align="left"><p>0.86</p></td><td align="left"><p>− 0.02</p></td><td char="." align="char"><p>0.07</p></td><td align="left"><p>− 0.02</p></td><td char="." align="char"><p>0.07</p></td><td char="." align="char"><p>0.35</p></td></tr><tr><td align="left" colspan="9"><p>Standardized total effects</p></td></tr><tr><td align="left"><p> AW → HOTS</p></td><td align="left"><p>− 0.02</p></td><td char="." align="char"><p>0.10</p></td><td align="left"><p>− 0.15</p></td><td align="left"><p>− 0.22</p></td><td char="." align="char"><p>0.18</p></td><td align="left"><p>− 0.22</p></td><td char="." align="char"><p>0.18</p></td><td char="." align="char"><p>0.87</p></td></tr><tr><td align="left"><p> ET → HOTS</p></td><td align="left"><p>0.09</p></td><td char="." align="char"><p>0.06</p></td><td align="left"><p>1.60</p></td><td align="left"><p>− 0.02</p></td><td char="." align="char"><p>0.21</p></td><td align="left"><p>− 0.03</p></td><td char="." align="char"><p>0.21</p></td><td char="." align="char"><p>0.13</p></td></tr><tr><td align="left"><p> US → HOTS</p></td><td align="left"><p>− 0.01</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>− 0.16</p></td><td align="left"><p>− 0.17</p></td><td char="." align="char"><p>0.13</p></td><td align="left"><p>− 0.17</p></td><td char="." align="char"><p>0.14</p></td><td char="." align="char"><p>0.88</p></td></tr><tr><td align="left"><p> EV → HOTS</p></td><td align="left"><p>0.53</p></td><td char="." align="char"><p>0.10</p></td><td align="left"><p>5.62</p></td><td align="left"><p>0.35</p></td><td char="." align="char"><p>0.72</p></td><td align="left"><p>0.35</p></td><td char="." align="char"><p>0.72</p></td><td char="." align="char"><p>0.000(**)</p></td></tr><tr><td align="left"><p> BE → HOTS</p></td><td align="left"><p>0.44</p></td><td char="." align="char"><p>0.14</p></td><td align="left"><p>3.14</p></td><td align="left"><p>0.20</p></td><td char="." align="char"><p>0.76</p></td><td align="left"><p>0.18</p></td><td char="." align="char"><p>0.73</p></td><td char="." align="char"><p>0.004(**)</p></td></tr><tr><td align="left"><p> PI → HOTS</p></td><td align="left"><p>0.28</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>3.34</p></td><td align="left"><p>0.12</p></td><td char="." align="char"><p>0.45</p></td><td align="left"><p>0.12</p></td><td char="." align="char"><p>0.45</p></td><td char="." align="char"><p>0.001(**)</p></td></tr><tr><td align="left"><p> AW → BE</p></td><td align="left"><p>0.13</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>1.59</p></td><td align="left"><p>− 0.05</p></td><td char="." align="char"><p>0.28</p></td><td align="left"><p>− 0.03</p></td><td char="." align="char"><p>0.29</p></td><td char="." align="char"><p>0.11</p></td></tr><tr><td align="left"><p> ET → BE</p></td><td align="left"><p>0.10</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>1.20</p></td><td align="left"><p>− 0.06</p></td><td char="." align="char"><p>0.26</p></td><td align="left"><p>− 0.07</p></td><td char="." align="char"><p>0.25</p></td><td char="." align="char"><p>0.27</p></td></tr><tr><td align="left"><p> US → BE</p></td><td align="left"><p>0.40</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>4.98</p></td><td align="left"><p>0.24</p></td><td char="." align="char"><p>0.55</p></td><td align="left"><p>0.24</p></td><td char="." align="char"><p>0.56</p></td><td char="." align="char"><p>0.000(**)</p></td></tr><tr><td align="left"><p> EV → BE</p></td><td align="left"><p>0.41</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>5.31</p></td><td align="left"><p>0.27</p></td><td char="." align="char"><p>0.57</p></td><td align="left"><p>0.25</p></td><td char="." align="char"><p>0.55</p></td><td char="." align="char"><p>0.00</p></td></tr><tr><td align="left"><p> AW → PI</p></td><td align="left"><p>0.17</p></td><td char="." align="char"><p>0.09</p></td><td align="left"><p>1.98</p></td><td align="left"><p>− 0.02</p></td><td char="." align="char"><p>0.33</p></td><td align="left"><p>− 0.01</p></td><td char="." align="char"><p>0.33</p></td><td char="." align="char"><p>0.07</p></td></tr><tr><td align="left"><p> ET → PI</p></td><td align="left"><p>0.06</p></td><td char="." align="char"><p>0.07</p></td><td align="left"><p>0.86</p></td><td align="left"><p>− 0.08</p></td><td char="." align="char"><p>0.20</p></td><td align="left"><p>− 0.09</p></td><td char="." align="char"><p>0.19</p></td><td char="." align="char"><p>0.42</p></td></tr><tr><td align="left"><p> US → PI</p></td><td align="left"><p>0.30</p></td><td char="." align="char"><p>0.09</p></td><td align="left"><p>3.47</p></td><td align="left"><p>0.13</p></td><td char="." align="char"><p>0.47</p></td><td align="left"><p>0.13</p></td><td char="." align="char"><p>0.47</p></td><td char="." align="char"><p>0.002(**)</p></td></tr><tr><td align="left"><p> EV → PI</p></td><td align="left"><p>0.30</p></td><td char="." align="char"><p>0.08</p></td><td align="left"><p>3.84</p></td><td align="left"><p>0.16</p></td><td char="." align="char"><p>0.47</p></td><td align="left"><p>0.14</p></td><td char="." align="char"><p>0.45</p></td><td char="." align="char"><p>0.001(**)</p></td></tr></tbody></table> </ephtml> </p> <p> <emph>EV</emph> Evaluation, <emph>US</emph> Usage, <emph>ET</emph> Ethics, <emph>AW</emph> Awareness, <emph>PI</emph> Peer interaction, <emph>BE</emph> Behavioral engagement, <emph>HOTS</emph> Higher order thinking skills *p < 0.05; **p < 0.01</p> <hd id="AN0185099437-39">Discussion and conclusion</hd> <p>The present study provides valuable insights into the relationship between AI literacy and HOTS, in a higher education context, particularly highlighting the mediating roles of behavioral engagement and peer interaction. Results of the study show that there are both direct and indirect relationship between AI literacy and HOTS.</p> <p>First, the relationship between the two process factors and HOTS was discussed. It was found that both behavioral engagement and peer interaction had significantly direct influences on HOTS. This finding is consistent with those reported in related studies (Eseryel et al., [<reflink idref="bib16" id="ref94">16</reflink>]; Guo et al., [<reflink idref="bib22" id="ref95">22</reflink>]; Lein et al., [<reflink idref="bib37" id="ref96">37</reflink>]). This finding can be attributed to the facets of the students' behavioral engagement domain, which encompasses classroom conduct, inquiry, participation in school-related activities, and interest in academic tasks (Nguyen et al., [<reflink idref="bib51" id="ref97">51</reflink>]). This is beneficial for HOTS development. Additionally, when students engage in peer interaction, they are exposed to diverse perspectives and are often encouraged to articulate and defend their ideas. This social aspect of learning promotes deeper cognitive processing, fostering skills such as critical analysis, information synthesis, and creative problem-solving—key components of HOTS. This finding indicated that instructors should endeavor to enhance students' behavioral engagement and peer interaction to promote college students' HOTS in AI era (Davenport & Ronanki, [<reflink idref="bib12" id="ref98">12</reflink>]; Kim et al., [<reflink idref="bib33" id="ref99">33</reflink>]). For example, researchers could delve deeper into understanding how specific aspects of behavioral engagement and peer interaction, such as inquiry-based learning, problem-based learning, or the use of AI-mediated peer collaboration, could contribute to the development of HOTS.</p> <p>Second, the influences of each dimension of AI literacy on process factors and HOTS were discussed. Referring to the influences of AI awareness on process factors and HOTS, it was found that AI awareness had no direct influence on behavioral engagement and HOTS. This finding is not consistent with those reported in related studies (Chen et al., [<reflink idref="bib7" id="ref100">7</reflink>]; Li et al., [<reflink idref="bib40" id="ref101">40</reflink>]). This may be because AI awareness alone does not prompt students to integrate AI technologies into their academic tasks unless they are explicitly taught how to apply these tools to enhance learning. Without active application, AI awareness remains a passive understanding, offering no substantial advantage in fostering engagement or HOTS. This suggests that AI awareness must be accompanied by practical skills and opportunities to use AI in real-world learning contexts to have measurable impacts on engagement and cognitive outcomes.</p> <p>But it was found that AI awareness had indirect effect on HOTS via the mediating variable of peer interaction. This finding highlights the significance of the social dimension of learning in the context of AI. While students with higher AI awareness may not engage with AI technologies independently, they are more likely to discuss AI-related topics with peers. Such discussions promote peer interaction, which positively influences HOTS. In other words, AI awareness may act as a catalyst for peer-based learning, where students exchange ideas, collaborate, and reflect on AI applications, thereby cultivating their HOTS. This suggests that educators should encourage collaborative AI-related activities and discussions to maximize the benefits of AI awareness on student learning outcomes.</p> <p>Referring to the influences of AI usage on process factors and HOTS, it was found that AI usage had significant and positive influence on behavioral engagement and peer interaction. This finding is consistent with those reported by scholars (Li et al., [<reflink idref="bib40" id="ref102">40</reflink>]; Prior et al., [<reflink idref="bib54" id="ref103">54</reflink>]). However, it was surprising and interesting to observe a significant and negative correlation between AI usage and HOTS. This can be attributed to the fact that, during high school, students were restricted from using smartphones and laptops in classes, barring the information technology class. Consequently, they had limited exposure to AI applications. Upon entering college, where they faced no such restrictions, they had sudden access to these technologies. This abrupt transition might have led students to misunderstand the correct approach towards AI applications, potentially leading to an overreliance or dependence on AI for problem-solving. When students rely too heavily on AI, their own cognitive efforts diminish. The development of HOTS necessitates robust analyzing, evaluating, and creating capabilities (Lu et al., [<reflink idref="bib43" id="ref104">43</reflink>], [<reflink idref="bib44" id="ref105">44</reflink>]). Hence, an overdependence on AI applications can hinder the cultivation of HOTS. This observation might also support the rationale behind several countries' decision to restrict ChatGPT in educational settings (Choi et al., [<reflink idref="bib10" id="ref106">10</reflink>]). Given that young individuals are still shaping their ideologies, unchecked AI app usage, without proper guidance, could negatively influence them. This finding suggests that AI tools should be integrated thoughtfully, with an emphasis on balancing technology use with opportunities for students to engage in meaningful cognitive effort, such as analysis, critical thinking, and problem-solving.</p> <p>When considering the relationship between AI evaluation on process factors and HOTS, we found that AI evaluation had significant and positive influences on behavioral engagement, peer interaction and HOTS. This finding is consistent with those reported by previous studies (Li et al., [<reflink idref="bib40" id="ref107">40</reflink>]; Prior et al., [<reflink idref="bib54" id="ref108">54</reflink>]). Since AI evaluation refers to the ability to select, analyze and critically evaluate AI applications and their results, evaluation is originally a sub-dimension of higher order thinking skills (Hwang et al., [<reflink idref="bib28" id="ref109">28</reflink>]). When students evaluate an AI app, they first need to query the information to understand which indicators to evaluate based on, and then they need to compare different parameters of the same indicator in different AI apps. In other words, evaluation itself is a process of training higher-order thinking skills. This result underscores the importance of integrating AI evaluation tasks into the curriculum to foster both student engagement and cognitive development. By actively evaluating AI applications, students move beyond passive technology use, becoming critical users who can assess the strengths and limitations of AI tools. This approach not only enhances their higher-order thinking skills but also prepares them for the complex decision-making processes they will encounter in both academic and professional contexts.</p> <p>Referring to the influences of AI ethics on process factors and HOTS, it was found that AI ethics had no significant influence on behavioral engagement, peer interaction and HOTS. This finding is inconsistent with those obtained in previous studies (Chen et al., [<reflink idref="bib7" id="ref110">7</reflink>]; Prior et al., [<reflink idref="bib54" id="ref111">54</reflink>]). This can be understood by considering that AI ethics pertains to the awareness of the responsibilities and risks related to the usage of AI technologies. Whether a student uses AI doesn't necessarily reflect their sense of responsibility or risk awareness; they might simply be conscious of aspects like privacy and security. Such considerations don't appear to influence behavioral engagement, peer interaction, or HOTS. This results may also be attributed to the fact that, while ethical awareness is valuable, it has not been embedded in practical learning scenarios that require students to apply ethical principles to complex problem-solving. Without applying ethical principles to real-world or problem-based learning tasks, students may not encounter the cognitive challenge necessary to foster HOTS. Therefore, it is essential to explore ways to integrate AI ethics more effectively into collaborative and problem-based learning environments, ensuring that ethical considerations become a core part of students' critical thinking and decision-making processes.</p> <hd id="AN0185099437-40">Theoretical and pedagogical value</hd> <p>From the theoretical perspective, this study verified that the 3P model is a great theoretical framework to understand the relationship between AI literacy and HOTS in higher education context. In addition, the mediating role of behavioral engagement and peer interaction was also investigated. Findings of the study provided a more integrative perspective on how to balance the relationship between AI literacy and HOTS.</p> <p>From the pedagogical perspective, this study offers valuable insights. Firstly, in terms of AI awareness, it is necessary to guide college students towards an accurate understanding of AI—as a supplementary tool rather than a replacement for our cognitive processes. For example, leveraging AI for routine and repetitive tasks can free up time, allowing us to tackle more complex, higher-order challenges, which in turn nurtures HOTS. Second, in terms of AI usage, it is necessary to guide college students to use AI rationally and efficiently. As the old saying goes, "everyone going too far is as bad as not going far enough." While AI technologies can be invaluable when required, it's essential not to become wholly dependent on them for every aspect. Third, in terms of AI evaluation, it is necessary to evaluate AI applications from a critical perspective, so that the right tools can be selected. On the other hand, it is conducive to cultivating higher order thinking skills. Forth, in terms of AI ethics, although AI ethics has no impact on behavioral engagement, peer interaction, and HOTS, it is still necessary for students to pay attention to ethical aspects, their privacy and security when using AI apps. Finally, we need to encourage college students to improve their behavioral engagement and peer interaction in the learning process, since both of them had positive influences on HOTS. For example, instructors could design some collaborative inquiry-based learning activities, so that students are able to communicate and collaborate with others to complete learning tasks. This will be helpful to cultivate students' behavioral engagement and peer interaction and may positively impact HOTS (Lu et al., [<reflink idref="bib47" id="ref112">47</reflink>]). Additionally, some incentive mechanisms can be introduced into classroom teaching, including immediate evaluation, feedback and rewards (Zhong & Xia, [<reflink idref="bib68" id="ref113">68</reflink>]).</p> <hd id="AN0185099437-41">Limitations and future research directions</hd> <p>While the present study has important implications, some limitations still should be acknowledged. Firstly, it should be noticed that we have incorporated two important learning factors (behavioral engagement and peer interaction) as the learning process factors into the structural model. Some other relevant factors, such as emotional engagement, self-regulated learning and cognitive engagement can also be considered by researchers in future studies. These elements may provide a more comprehensive understanding of the learning process and its influence between AI literacy and higher-order thinking skills.</p> <p>Secondly, data used in the present study was self-reported, i.e. subjective. Some other objective evidence (e.g. learning behavior observations and analyses) can be considered in future studies to triangulate findings with self-reported data. Specifically, methods like social network analysis could be employed to examine peer interaction patterns in a more nuanced way. Researchers could quantitatively assess the structure and quality of peer networks and their mediating influences between AI literacy and higher order thinking skills.</p> <p>Third, the quantitative approach was used in the study, which does not allow in-depth analysis of the relationships between AI literacy and HOTS. Further studies should take a more qualitative approach to gain a more thorough and deeper understanding of this issue. A qualitative approach, such as in-depth interviews with students could be conducted. Interviews would allow researchers to explore students' perceptions, motivations, and personal learning experiences in greater depth, uncovering factors that are not easily captured by quantitative measures. This method would offer a more comprehensive understanding of how individual students navigate learning processes and apply AI literacy in different contexts.</p> <p>Lastly, future studies should aim for a broader and more diverse participant pool, e.g. expanding it beyond university-level students such as learners from elementary or secondary schools. Involving younger students may provide insights into how AI literacy and higher-order thinking skills develop at different stages of education, helping to inform age-appropriate pedagogical approaches. Including participants from various socio-cultural and educational backgrounds will also enhance the generalizability of the findings, offering a more holistic view of how different populations engage with AI literacy and HOTS across educational settings.</p> <hd id="AN0185099437-42">Acknowledgements</hd> <p>This work was supported by the Major Project of Philosophy and Social Science Research in Colleges and Universities of Jiangsu Province, "Research on hierarchical collaborative inquiry-based teaching styles based on the smart education platform of primary and secondary schools in Jiangsu" (Grant No. 2024SJZD074), the Education Science Planning Project of Jiangsu Province, "Research on the mechanism and intervention of AI-supported collective intelligence in collaborative inquiry-based learning" (Grant No. C/2023/01/108), the National Natural Science Foundation of China (Grant No. 62407023, 62407008), the Jiangsu Social Science Fund, "Research on the legalization of corruption governance enabled by digital governance" (Grant No. 22FXB001).</p> <hd id="AN0185099437-43">Data availability</hd> <p>The datasets generated and analyzed during the current study are not publicly available but will be provided by the corresponding author on reasonable request.</p> <hd id="AN0185099437-44">Declarations</hd> <p></p> <hd id="AN0185099437-45">Competing interests</hd> <p>The authors declare no competing interests.</p> <hd id="AN0185099437-46">Appendix</hd> <p>Participants' AI literacy, behavioral engagement, peer interaction, and higher order thinking skills are assessed based on specific criteria. Participants rate each item on a 5-point Likert-type scale, which ranges from 1 (strongly disagree) to 5 (strongly agree). Below is a breakdown of these criteria:</p> <hd id="AN0185099437-47">AI literacy</hd> <p></p> <hd id="AN0185099437-48">Awareness</hd> <p></p> <ulist> <item> I can distinguish between smart devices and non-smart devices.</item> <p></p> <item> I know how AI technology can help me.</item> <p></p> <item> I can identify the AI technology employed in the applications and products I use.</item> </ulist> <hd id="AN0185099437-49">Usages</hd> <p></p> <ulist> <item> I can skillfully use AI applications or products to help me with my daily work.</item> <p></p> <item> It is usually easy for me to learn to use a new AI application or product.</item> <p></p> <item> I can use AI applications or products to improve my work efficiency.</item> </ulist> <hd id="AN0185099437-50">Evaluation</hd> <p></p> <ulist> <item> I can evaluate the capabilities and limitations of an AI application or product after using it for a while.</item> <p></p> <item> I can choose a proper solution from various solutions provided by a smart agent.</item> <p></p> <item> I can choose the most appropriate AI application or product from a variety for a particular task.</item> </ulist> <hd id="AN0185099437-51">Ethics</hd> <p></p> <ulist> <item> I always comply with ethical principles when using AI applications or products.</item> <p></p> <item> I am alert to privacy and information security issues when using AI applications or products.</item> <p></p> <item> I am always alert to the abuse of AI technology.</item> </ulist> <hd id="AN0185099437-52">Behavioral engagement</hd> <p></p> <ulist> <item> I try hard to do well in all of the learning activities.</item> <p></p> <item> In learning, I work as hard as I can.</item> <p></p> <item> In learning, I participate in all the learning activities.</item> </ulist> <hd id="AN0185099437-53">Peer interaction</hd> <p></p> <hd id="AN0185099437-54">Communication</hd> <p></p> <ulist> <item> I try to make the other person feel good.</item> <p></p> <item> I try to make the other person feel important.</item> <p></p> <item> I try to be warm when communicating with others.</item> <p></p> <item> While I'm talking I think about how the other person feels.</item> <p></p> <item> I am verbally and nonverbally supportive of other people.</item> <p></p> <item> I disclose at the same level that others disclose to me.</item> </ulist> <hd id="AN0185099437-55">Collaboration</hd> <p></p> <ulist> <item> I believe our team can cooperate successfully when I conduct collaborative learning.</item> <p></p> <item> I try to provide useful and sufficient information when I conduct collaborative learning.</item> <p></p> <item> I have good communication with my team members when I conduct collaborative learning.</item> <p></p> <item> I can finish my work efficiently when I conduct collaborative learning.</item> <p></p> <item> Work is split based on our abilities when I conduct collaborative learning.</item> </ulist> <hd id="AN0185099437-56">Higher order thinking skills</hd> <p></p> <hd id="AN0185099437-57">Problem-solving</hd> <p></p> <ulist> <item> 1.When facing problems, I believe I have the ability to solve them.</item> <p></p> <item> I believe I can put effort into solving problems.</item> <p></p> <item> I can solve problems that I have met before.</item> <p></p> <item> I am willing to face problems and make an effort to solve them.</item> </ulist> <hd id="AN0185099437-58">Critical thinking</hd> <p></p> <ulist> <item> I ask myself periodically if I am meeting my goals.</item> <p></p> <item> I consider several alternatives to a problem before I answer.</item> <p></p> <item> I find myself pausing regularly to check my comprehension.</item> <p></p> <item> I ask myself questions about how well I am doing once I finish a task.</item> </ulist> <hd id="AN0185099437-59">Creativity</hd> <p></p> <ulist> <item> I like to observe something I haven't seen before and understand it in detail.</item> <p></p> <item> I like to try something new.</item> <p></p> <item> I like to do something by myself.</item> </ulist> <hd id="AN0185099437-60">Publisher's Note</hd> <p>Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.</p> <ref id="AN0185099437-61"> <title> References </title> <blist> <bibl id="bib1" idref="ref92" type="bt">1</bibl> <bibtext> Arnold KA, Connelly CE, Walsh MM, Martin Ginis KA. Leadership styles, emotion regulation, and burnout. Journal of Occupational Health Psychology. 2015; 20; 4: 481. 10.1037/a0039045</bibtext> </blist> <blist> <bibl id="bib2" idref="ref8" type="bt">2</bibl> <bibtext> Azevedo R, Bouchet F, Duffy M, Harley J, Taub M, Trevors G, Cerezo R. Lessons learned and future directions of MetaTutor: Leveraging multichannel data to scaffold self-regulated learning with an intelligent tutoring system. Frontiers in Psychology. 2022; 13: 813632. 10.3389/fpsyg.2022.813632</bibtext> </blist> <blist> <bibl id="bib3" idref="ref24" type="bt">3</bibl> <bibtext> Biggs J. What do inventories of students' learning processes really measure? A theoretical review and clarification. British Journal of Educational Psychology. 1993; 63; 1: 3-19. 10.1111/j.2044-8279.1993.tb01038.x</bibtext> </blist> <blist> <bibl id="bib4" idref="ref33" type="bt">4</bibl> <bibtext> Biggs J, Kember D, Leung DY. The revised two-factor study process questionnaire: R-SPQ-2F. British Journal of Educational Psychology. 2001; 71; 1: 133-149. 10.1348/000709901158433</bibtext> </blist> <blist> <bibl id="bib5" idref="ref45" type="bt">5</bibl> <bibtext> Burgsteiner H, Kandlhofer M, Steinbauer G. Irobot: Teaching the basics of artificial intelligence in high schools. In Proceedings of the AAAI Conference on Artificial Intelligence. 2016. 10.1609/aaai.v30i1.9864</bibtext> </blist> <blist> <bibl id="bib6" idref="ref81" type="bt">6</bibl> <bibtext> Celik I. Exploring the determinants of artificial intelligence (Ai) literacy: Digital divide, computational thinking, cognitive absorption. Telematics and Informatics. 2023; 83. 10.1016/j.tele.2023.102026102026</bibtext> </blist> <blist> <bibl id="bib7" idref="ref100" type="bt">7</bibl> <bibtext> Chen LC, Chen YH, Ma WI. Effects of integrated information literacy on science learning and problem-solving among seventh-grade students. Malaysian Journal of Library & Information Science. 2014; 19; 2: 35</bibtext> </blist> <blist> <bibl id="bib8" idref="ref57" type="bt">8</bibl> <bibtext> Chen X, Xie H, Hwang GJ. A multi-perspective study on artificial intelligence in education: Grants, conferences, journals, software tools, institutions, and researchers. Computers and Education: Artificial Intelligence. 2020; 1100005</bibtext> </blist> <blist> <bibl id="bib9" idref="ref83" type="bt">9</bibl> <bibtext> Chiu TK. Digital support for student engagement in blended learning based on self-determination theory. Computers in Human Behavior. 2021; 124. 10.1016/j.chb.2021.106909106909</bibtext> </blist> <blist> <bibtext> Choi JH, Hickman KE, Monahan A, Schwarcz D. ChatGPT goes to law school. SSRN Electronic Journal. 2023. 10.2139/ssrn.4335905</bibtext> </blist> <blist> <bibtext> Christudason, A. (n. d.). What is peer interaction/learning. Retrieved August 4, 2023, from What is Peer Interaction/Learning | IGI Global (igi-global.com).</bibtext> </blist> <blist> <bibtext> Davenport TH, Ronanki R. Artificial intelligence for the real world. Harvard Business Review. 2018; 96; 1: 108-116</bibtext> </blist> <blist> <bibtext> Delcker J, Heil J, Ifenthaler D, Seufert S, Spirgi L. First-year students AI-competence as a predictor for intended and de facto use of AI-tools for supporting learning processes in higher education. International Journal of Educational Technology in Higher Education. 2024; 21: 18. 10.1186/s41239-024-00452-7</bibtext> </blist> <blist> <bibtext> Dunkin MJ, Biddle BJ. The study of teaching. 1974; Holt, Rinehart & Winston</bibtext> </blist> <blist> <bibtext> Dwivedi YK, Kshetri N, Hughes L, Slade EL, Jeyaraj A, Kar AK, Wright R. "So what if ChatGPT wrote it?" Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management. 2023; 71: 102642. 10.1016/j.ijinfomgt.2023.102642</bibtext> </blist> <blist> <bibtext> Eseryel D, Law V, Ifenthaler D, Ge X, Miller R. An investigation of the interrelationships between motivation, engagement, and complex problem solving in game-based learning. Journal of Educational Technology & Society. 2014; 17; 1: 42-53</bibtext> </blist> <blist> <bibtext> Eshet Y. Digital literacy: A conceptual framework for survival skills in the digital era. Journal of Educational Multimedia and Hypermedia. 2004; 13; 1: 93-106</bibtext> </blist> <blist> <bibtext> Fornell C, Larcker DF. Evaluating Structural Equation Models with unobservable variables and measurement error. Journal of Marketing Research. 1981; 18; 1: 39-50. 10.1177/002224378101800104</bibtext> </blist> <blist> <bibtext> Frazier K, Reynolds E. Power up your creative mind. 2012; Pieces of Learning</bibtext> </blist> <blist> <bibtext> Fredricks JA, Blumenfeld PC, Paris AH. School engagement: Potential of the concept, state of the evidence. Review of Educational Research. 2004; 74; 1: 59-109. 10.3102/00346543074001059</bibtext> </blist> <blist> <bibtext> Gunkel DJ. The machine question: Critical perspectives on AI, robots, and ethics. 2012; Mit Press. 10.7551/mitpress/8975.001.0001</bibtext> </blist> <blist> <bibtext> Guo F, Yao M, Wang C, Yan W, Zong X. The effects of service learning on student problem solving: The mediating role of classroom engagement. Teaching of Psychology. 2016; 43; 1: 16-21. 10.1177/0098628315620064</bibtext> </blist> <blist> <bibtext> Hair JFJ, Black WC, Babin BJ, Anderson RE. Multivariate data analysis: A global perspective. 20107; Pearson</bibtext> </blist> <blist> <bibtext> Hallaq T. Evaluating online media literacy in higher education: Validity and reliability of the digital online media literacy assessment (DOMLA). Journal of Media Literacy Education. 2016; 8; 1: 62-84</bibtext> </blist> <blist> <bibtext> Han JH. Closing the missing links and opening the relationships among the factors: A literature review on the use of clicker technology using the 3P model. Journal of Educational Technology & Society. 2014; 17; 4: 150-168</bibtext> </blist> <blist> <bibtext> Huang W, Roscoe RD, Johnson-Glenberg MC, Craig SD. Motivation, engagement, and performance across multiple virtual reality sessions and levels of immersion. Journal of Computer Assisted Learning. 2021; 37; 3: 745-758. 10.1111/jcal.12520</bibtext> </blist> <blist> <bibtext> Huang YM, Silitonga LM, Wu TT. Applying a business simulation game in a flipped classroom to enhance engagement, learning achievement, and higher-order thinking skills. Computers & Education. 2022; 183. 10.1016/j.compedu.2022.104494104494</bibtext> </blist> <blist> <bibtext> Hwang GJ, Lai CL, Liang JC, Chu HC, Tsai CC. A long-term experiment to investigate the relationships between high school students' perceptions of mobile learning and peer interaction and higher-order thinking tendencies. Educational Technology Research and Development. 2018; 66: 75-93. 10.1007/s11423-017-9540-3</bibtext> </blist> <blist> <bibtext> Ibna Seraj PM, Oteir I. Playing with AI to investigate human-computer interaction technology and improving critical thinking skills to pursue 21st century age. Education Research International. 2022. 10.1155/2022/6468995</bibtext> </blist> <blist> <bibtext> Jansen T, Möller J. Teacher judgments in school exams: Influences of students' lower-order-thinking skills on the assessment of students' higher-order-thinking skills. Teaching and Teacher Education. 2022; 111. 10.1016/j.tate.2021.103616103616</bibtext> </blist> <blist> <bibtext> Jones-Jang SM, Mortensen T, Liu J. Does media literacy help identification of fake news? Information literacy helps, but other literacies don't. American Behavioral Scientist. 2021; 65; 2: 371-388. 10.1177/0002764219869406</bibtext> </blist> <blist> <bibtext> Kandlhofer, M, Steinbauer, G, Hirschmugl-Gaisch, S, & Huber, P. (2016). Artificial intelligence and computer science in education: From kindergarten to university. In 2016 IEEE Frontiers in Education Conference (FIE) (pp. 1–9). IEEE.</bibtext> </blist> <blist> <bibtext> Kim HJ, Yi P, Hong JI. Students' academic use of mobile technology and higher-order thinking skills: The role of active engagement. Education Sciences. 2020; 10; 3: 47. 10.3390/educsci10030047</bibtext> </blist> <blist> <bibtext> Lane ES, Harris SE. A new tool for measuring student behavioral engagement in large university classes. Journal of College Science Teaching. 2015; 44; 6: 83-91. 10.2505/4/jcst15_044_06_83</bibtext> </blist> <blist> <bibtext> Laupichler MC, Aster A, Schirch J, Raupach T. Artificial intelligence literacy in higher and adult education: A scoping literature review. Computers and Education: Artificial Intelligence. 2022; 3: 100101</bibtext> </blist> <blist> <bibtext> Lee MH, Liang JC, Wu YT, Chiou GL, Hsu CY, Wang CY, Tsai CC. High school students' conceptions of science laboratory learning, perceptions of the science laboratory environment, and academic self-efficacy in science learning. International Journal of Science and Mathematics Education. 2020; 18: 1-18. 10.1007/s10763-019-09951-w</bibtext> </blist> <blist> <bibtext> Lein AE, Jitendra AK, Starosta KM, Dupuis DN, Hughes-Reid CL, Star JR. Assessing the relation between seventh-grade students' engagement and mathematical problem solving performance. Preventing School Failure: Alternative Education for Children and Youth. 2016; 60; 2: 117-123. 10.1080/1045988X.2015.1036392</bibtext> </blist> <blist> <bibtext> Lewis A, Smith D. Defining higher order thinking. Theory into Practice. 1993; 32; 3: 131-137. 10.1080/00405849309543588</bibtext> </blist> <blist> <bibtext> Li H. Effects of a ChatGPT-based flipped learning guiding approach on learners' courseware project performances and perceptions. Australasian Journal of Educational Technology. 2023; 39; 5: 40-58. 10.14742/ajet.8923</bibtext> </blist> <blist> <bibtext> Li H, Zhu S, Wu D, Yang HH, Guo Q. Impact of information literacy, self-directed learning skills, and academic emotions on high school students' online learning engagement: A structural equation modeling analysis. Education and information technologies. 2023. 10.1007/s10639-023-11760-2</bibtext> </blist> <blist> <bibtext> Li LY, Tsai CC. Students' patterns of accessing time in a text structure learning system: Relationship to individual characteristics and learning performance. Educational Technology Research and Development. 2020; 68: 2569-2594. 10.1007/s11423-020-09780-7</bibtext> </blist> <blist> <bibtext> Long, D, & Magerko, B. (2020, April). What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1–16).</bibtext> </blist> <blist> <bibtext> Lu K, Pang F, Shadiev R. Understanding the mediating effect of learning approach between learning factors and higher order thinking skills in collaborative inquiry-based learning. Educational Technology Research and Development. 2021; 69; 5: 2475-2492. 10.1007/s11423-021-10025-4</bibtext> </blist> <blist> <bibtext> Lu K, Pang F, Shadiev R. How to deepen college students' approach to using technologies in T-O-IBL? Examining the mediating influence of deep approaches to using technologies between learning factors and higher order thinking skills. Journal of Computer Assisted Learning. 2023; 39; 1: 182-193. 10.1111/jcal.12738</bibtext> </blist> <blist> <bibtext> Lu K, Shi Y, Li J, Yang HH, Xu M. An investigation of college students' learning engagement and classroom preferences under the smart classroom environment. SN Computer Science. 2022; 3; 3: 205. 10.1007/s42979-022-01093-1</bibtext> </blist> <blist> <bibtext> Lu K, Yang HH, Shi Y, Wang X. Examining the key influencing factors on college students' higher-order thinking skills in the smart classroom environment. International Journal of Educational Technology in Higher Education. 2021; 18: 1-13. 10.1186/s41239-020-00238-7</bibtext> </blist> <blist> <bibtext> Lu K, Yang H, Xue H. Investigating the four-level inquiry continuum on college students' higher order thinking and peer interaction tendencies. International Journal of Innovation and Learning. 2021; 30; 3: 358-367. 10.1504/IJIL.2021.118192</bibtext> </blist> <blist> <bibtext> Ng DTK, Leung JKL, Chu SKW, Qiao MS. Conceptualizing AI literacy: An exploratory review. Computers and Education: Artificial Intelligence. 2021; 2100041</bibtext> </blist> <blist> <bibtext> Ng DTK, Su J, Leung JKL, Chu SKW. Artificial intelligence (AI) literacy education in secondary schools: A review. Interactive Learning Environments. 2023. 10.1080/10494820.2023.2255228</bibtext> </blist> <blist> <bibtext> Nguyen A, Kremantzis M, Essien A, Petrounias I, Hosseini S. Enhancing student engagement through artificial intelligence (AI): Understanding the basics, opportunities, and challenges. Journal of University Teaching and Learning Practice. 2024. 10.53761/caraaq92</bibtext> </blist> <blist> <bibtext> Nguyen TD, Cannata M, Miller J. Understanding student behavioral engagement: Importance of student interaction with peers and teachers. The Journal of Educational Research. 2018; 111; 2: 163-174. 10.1080/00220671.2016.1220359</bibtext> </blist> <blist> <bibtext> Nunnally JC, Bernstein IH. Psychometric theory. 1994; McGraw-Hill</bibtext> </blist> <blist> <bibtext> Preacher KJ, Hayes AF. Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Behavior Research Methods. 2008; 40; 3: 879-891. 10.3758/BRM.40.3.879</bibtext> </blist> <blist> <bibtext> Prior DD, Mazanov J, Meacheam D, Heaslip G, Hanson J. Attitude, digital literacy and self-efficacy: Flow-on effects for online learning behavior. The Internet and Higher Education. 2016; 29: 91-97. 10.1016/j.iheduc.2016.01.001</bibtext> </blist> <blist> <bibtext> Reddy P, Sharma B, Chaudhary K. Digital literacy: A review of literature. International Journal of Technoethics (IJT). 2020; 11; 2: 65-94. 10.4018/IJT.20200701.oa1</bibtext> </blist> <blist> <bibtext> Segars AH. Assessing the unidimensionality of measurement: A paradigm and illustration within the context of information systems research. Omega International Journal of Management Science. 1997; 25; 1: 107-121. 10.1016/S0305-0483(96)00051-5</bibtext> </blist> <blist> <bibtext> Shadiev R, Zhang ZH, Wu TT, Huang YM. Review of studies on recognition technologies and their applications used to assist learning and instruction. Educational Technology & Society. 2020; 23; 4: 59-74</bibtext> </blist> <blist> <bibtext> Skinner EA, Kindermann TA, Furrer CJ. A motivational perspective on engagement and disaffection: Conceptualization and assessment of children's behavioral and emotional participation in academic activities in the classroom. Educational and Psychological Measurement. 2009; 69; 3: 493-525. 10.1177/0013164408323233</bibtext> </blist> <blist> <bibtext> Southworth J, Migliaccio K, Glover J, Reed D, McCarty C, Brendemuhl J, Thomas A. Developing a model for AI Across the curriculum: Transforming the higher education landscape via innovation in AI literacy. Computers and Education: Artificial Intelligence. 2023; 4100127</bibtext> </blist> <blist> <bibtext> Sun JCY, Rueda R. Situational interest, computer self-efficacy and self-regulation: Their impact on student engagement in distance education. British Journal of Educational Technology. 2012; 43; 2: 191-204. 10.1111/j.1467-8535.2010.01157.x</bibtext> </blist> <blist> <bibtext> Tugtekin EB, Koc M. Understanding the relationship between new media literacy, communication skills, and democratic tendency: Model development and testing. New Media & Society. 2020; 22; 10: 1922-1941. 10.1177/1461444819887705</bibtext> </blist> <blist> <bibtext> van den Berg G, du Plessis E. ChatGPT and generative AI: Possibilities for its contribution to lesson planning, critical thinking and openness in teacher education. Education Sciences. 2023; 13; 10: 998. 10.3390/educsci13100998</bibtext> </blist> <blist> <bibtext> von Ende E, Ryan S, Crain MA, Makary MS. Artificial intelligence, augmented reality, and virtual reality advances and applications in interventional radiology. Diagnostics. 2023; 13; 5: 892. 10.3390/diagnostics13050892</bibtext> </blist> <blist> <bibtext> Wang B, Rau PLP, Yuan T. Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology. 2023; 42; 9: 1324-1337. 10.1080/0144929X.2022.2072768</bibtext> </blist> <blist> <bibtext> Wang L, Chen X, Wang C, Xu L, Shadiev R, Li Y. ChatGPT's capabilities in providing feedback on undergraduate students' argumentation: A case study. Thinking Skills and Creativity. 2024; 51. 10.1016/j.tsc.2023.101440101440</bibtext> </blist> <blist> <bibtext> Webber S, Johnston B. Conceptions of information literacy: New perspectives and implications. Journal of Information Science. 2000; 26; 6: 381-397. 10.1177/016555150002600602</bibtext> </blist> <blist> <bibtext> Zain FM, Sailin SN, Mahmor NA. Promoting higher order thinking skills among pre-service teachers through group-based flipped learning. International Journal of Instruction. 2022; 15; 3: 519-542. 10.29333/iji.2022.15329a</bibtext> </blist> <blist> <bibtext> Zhong B, Xia L. Effects of new coopetition designs on learning performance in robotics education. Journal of Computer Assisted Learning. 2022; 38; 1: 223-236. 10.1111/jcal.12606</bibtext> </blist> </ref> <aug> <p>By Kaili Lu; Jianrong Zhu; Feng Pang and Rustam Shadiev</p> <p>Reported by Author; Author; Author; Author</p> <p></p> <p>Kaili Lu Kaili Lu is a lecture of the College of Education Science and Technology at Nanjing University of Posts and Telecommunications, China. Her research interests include artificial intelligence, higher order thinking skills and technology-enhanced learning.</p> <p>Jianrong Zhu Jianrong Zhu is an associate researcher at Nanjing University of Posts and Telecommunications, China. His research interests include digital governance and legal theory and practice.</p> <p>Feng Pang Feng Pang is a doctor candidate in College of Education at De La Salle University-Dasmariñas, Philippines. His research interests include artificial intelligence and technology-enhanced learning.</p> <p>Rustam Shadiev Rustam Shadiev is a tenured professor at the College of Education, Zhejiang University, China. His research interests include technology-supported language learning and cross-cultural education.</p> </aug> <nolink nlid="nl1" bibid="bib27" firstref="ref1"></nolink> <nolink nlid="nl2" bibid="bib43" firstref="ref2"></nolink> <nolink nlid="nl3" bibid="bib46" firstref="ref3"></nolink> <nolink nlid="nl4" bibid="bib30" firstref="ref4"></nolink> <nolink nlid="nl5" bibid="bib44" firstref="ref5"></nolink> <nolink nlid="nl6" bibid="bib67" firstref="ref6"></nolink> <nolink nlid="nl7" bibid="bib38" firstref="ref7"></nolink> <nolink nlid="nl8" bibid="bib13" firstref="ref9"></nolink> <nolink nlid="nl9" bibid="bib15" firstref="ref10"></nolink> <nolink nlid="nl10" bibid="bib65" firstref="ref11"></nolink> <nolink nlid="nl11" bibid="bib12" firstref="ref12"></nolink> <nolink nlid="nl12" bibid="bib42" firstref="ref13"></nolink> <nolink nlid="nl13" bibid="bib35" firstref="ref14"></nolink> <nolink nlid="nl14" bibid="bib64" firstref="ref15"></nolink> <nolink nlid="nl15" bibid="bib62" firstref="ref16"></nolink> <nolink nlid="nl16" bibid="bib34" firstref="ref19"></nolink> <nolink nlid="nl17" bibid="bib51" firstref="ref20"></nolink> <nolink nlid="nl18" bibid="bib37" firstref="ref21"></nolink> <nolink nlid="nl19" bibid="bib54" firstref="ref22"></nolink> <nolink nlid="nl20" bibid="bib61" firstref="ref23"></nolink> <nolink nlid="nl21" bibid="bib14" firstref="ref25"></nolink> <nolink nlid="nl22" bibid="bib40" firstref="ref26"></nolink> <nolink nlid="nl23" bibid="bib41" firstref="ref27"></nolink> <nolink nlid="nl24" bibid="bib25" firstref="ref28"></nolink> <nolink nlid="nl25" bibid="bib36" firstref="ref29"></nolink> <nolink nlid="nl26" bibid="bib31" firstref="ref34"></nolink> <nolink nlid="nl27" bibid="bib66" firstref="ref35"></nolink> <nolink nlid="nl28" bibid="bib17" firstref="ref36"></nolink> <nolink nlid="nl29" bibid="bib55" firstref="ref37"></nolink> <nolink nlid="nl30" bibid="bib48" firstref="ref39"></nolink> <nolink nlid="nl31" bibid="bib57" firstref="ref40"></nolink> <nolink nlid="nl32" bibid="bib24" firstref="ref44"></nolink> <nolink nlid="nl33" bibid="bib32" firstref="ref46"></nolink> <nolink nlid="nl34" bibid="bib21" firstref="ref48"></nolink> <nolink nlid="nl35" bibid="bib45" firstref="ref49"></nolink> <nolink nlid="nl36" bibid="bib60" firstref="ref50"></nolink> <nolink nlid="nl37" bibid="bib20" firstref="ref51"></nolink> <nolink nlid="nl38" bibid="bib11" firstref="ref52"></nolink> <nolink nlid="nl39" bibid="bib28" firstref="ref53"></nolink> <nolink nlid="nl40" bibid="bib19" firstref="ref54"></nolink> <nolink nlid="nl41" bibid="bib63" firstref="ref58"></nolink> <nolink nlid="nl42" bibid="bib26" firstref="ref59"></nolink> <nolink nlid="nl43" bibid="bib59" firstref="ref60"></nolink> <nolink nlid="nl44" bibid="bib50" firstref="ref61"></nolink> <nolink nlid="nl45" bibid="bib49" firstref="ref64"></nolink> <nolink nlid="nl46" bibid="bib16" firstref="ref67"></nolink> <nolink nlid="nl47" bibid="bib22" firstref="ref70"></nolink> <nolink nlid="nl48" bibid="bib29" firstref="ref76"></nolink> <nolink nlid="nl49" bibid="bib39" firstref="ref78"></nolink> <nolink nlid="nl50" bibid="bib18" firstref="ref79"></nolink> <nolink nlid="nl51" bibid="bib58" firstref="ref82"></nolink> <nolink nlid="nl52" bibid="bib23" firstref="ref87"></nolink> <nolink nlid="nl53" bibid="bib52" firstref="ref88"></nolink> <nolink nlid="nl54" bibid="bib56" firstref="ref90"></nolink> <nolink nlid="nl55" bibid="bib53" firstref="ref93"></nolink> <nolink nlid="nl56" bibid="bib33" firstref="ref99"></nolink> <nolink nlid="nl57" bibid="bib10" firstref="ref106"></nolink> <nolink nlid="nl58" bibid="bib47" firstref="ref112"></nolink> <nolink nlid="nl59" bibid="bib68" firstref="ref113"></nolink>
Header DbId: eric
DbLabel: ERIC
An: EJ1470802
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Understanding the Relationship between Colleges Students' Artificial Intelligence Literacy and Higher Order Thinking Skills Using the 3P Model: The Mediating Roles of Behavioral Engagement and Peer Interaction
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Kaili+Lu%22">Kaili Lu</searchLink><br /><searchLink fieldCode="AR" term="%22Jianrong+Zhu%22">Jianrong Zhu</searchLink><br /><searchLink fieldCode="AR" term="%22Feng+Pang%22">Feng Pang</searchLink><br /><searchLink fieldCode="AR" term="%22Rustam+Shadiev%22">Rustam Shadiev</searchLink> (ORCID <externalLink term="http://orcid.org/0000-0001-5571-1158">0000-0001-5571-1158</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Educational+Technology+Research+and+Development%22"><i>Educational Technology Research and Development</i></searchLink>. 2025 73(2):693-716.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 24
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research<br />Tests/Questionnaires
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22College+Students%22">College Students</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Digital+Literacy%22">Digital Literacy</searchLink><br /><searchLink fieldCode="DE" term="%22Thinking+Skills%22">Thinking Skills</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Peer+Relationship%22">Peer Relationship</searchLink><br /><searchLink fieldCode="DE" term="%22Cooperative+Learning%22">Cooperative Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Interaction%22">Interaction</searchLink><br /><searchLink fieldCode="DE" term="%22Ethics%22">Ethics</searchLink><br /><searchLink fieldCode="DE" term="%22Learner+Engagement%22">Learner Engagement</searchLink><br /><searchLink fieldCode="DE" term="%22Curriculum+Design%22">Curriculum Design</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1007/s11423-024-10434-1
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1042-1629<br />1556-6501
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Artificial Intelligence (AI) has brought about significant changes in our lives, making AI literacy a crucial endeavor for the future. Despite its growing importance in academia, there is limited empirical research on its impact on college students' higher order thinking skills (HOTS). The present study systematically and comprehensively explores the relationship between college students' AI literacy and HOTS using the 3P (Presage-process-product) model. In this model, students' AI literacy represents the presage factors, while behavioral engagement and peer interaction serves as the process factors, and HOTS is the product factor. We gathered data from a survey of 260 college students. We utilized structural equation modeling to analyze the relationships between the 3P factors. The results showed that both AI usage and AI evaluation directly influenced HOTS and also indirectly affected HOTS through the mediating role of behavioral engagement and peer interaction. Conversely, AI awareness and AI ethics showed no direct influence on HOTS, although AI awareness impacted HOTS via peer interaction mediation. The results of the study have several theoretical and practical implications. From a theoretical perspective, this study incorporates AI literacy, behavioral engagement, peer interaction, and HOTS within the 3P model framework, shedding light on their interrelations. On a practical note, the results emphasize the need to consider AI literacy, behavioral engagement, peer interaction when designing courses to enhance HOTS in the era of AI.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1470802
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1470802
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1007/s11423-024-10434-1
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 24
        StartPage: 693
    Subjects:
      – SubjectFull: College Students
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Digital Literacy
        Type: general
      – SubjectFull: Thinking Skills
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Peer Relationship
        Type: general
      – SubjectFull: Cooperative Learning
        Type: general
      – SubjectFull: Interaction
        Type: general
      – SubjectFull: Ethics
        Type: general
      – SubjectFull: Learner Engagement
        Type: general
      – SubjectFull: Curriculum Design
        Type: general
    Titles:
      – TitleFull: Understanding the Relationship between Colleges Students' Artificial Intelligence Literacy and Higher Order Thinking Skills Using the 3P Model: The Mediating Roles of Behavioral Engagement and Peer Interaction
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Kaili Lu
      – PersonEntity:
          Name:
            NameFull: Jianrong Zhu
      – PersonEntity:
          Name:
            NameFull: Feng Pang
      – PersonEntity:
          Name:
            NameFull: Rustam Shadiev
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 04
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1042-1629
            – Type: issn-electronic
              Value: 1556-6501
          Numbering:
            – Type: volume
              Value: 73
            – Type: issue
              Value: 2
          Titles:
            – TitleFull: Educational Technology Research and Development
              Type: main
ResultId 1