Item Response Theory-Based Gaming Detection
Saved in:
| Title: | Item Response Theory-Based Gaming Detection |
|---|---|
| Language: | English |
| Authors: | Huang, Yun, Dang, Steven, Richey, J. Elizabeth, Asher, Michael, Lobczowski, Nikki G., Chine, Danielle, McLaughlin, Elizabeth A., Harackiewicz, Judith M., Aleven, Vincent, Koedinger, Kenneth |
| Source: | International Educational Data Mining Society. 2022. |
| Availability: | International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/ |
| Peer Reviewed: | Y |
| Page Count: | 12 |
| Publication Date: | 2022 |
| Document Type: | Speeches/Meeting Papers Reports - Descriptive |
| Education Level: | Junior High Schools Middle Schools Secondary Education High Schools |
| Descriptors: | Item Response Theory, Learner Engagement, Student Behavior, Student Motivation, Academic Achievement, Middle School Students, High School Students, Algebra, Learning Strategies |
| Abstract: | Gaming the system, a behavior in which learners exploit a system's properties to make progress while avoiding learning, has frequently been shown to be associated with lower learning. However, when we applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detected gaming was not associated with learning, challenging its construct validity. Our iterative exploratory data analysis suggested that some contextual factors that varied across and within conditions might contribute to this lack of association. We present a latent variable model, "item response theory-based gaming detection" (IRT-GD), that accounts for contextual factors and estimates latent gaming tendencies as the degree of deviation from normative behaviors across contexts. Item response theory models, widely used in knowledge assessment, account for item difficulty in estimating latent student abilities: students are estimated as having higher ability when they can get harder items correct than when they only get easier items correct. Similarly, IRT-GD accounts for contextual factors in estimating latent gaming tendencies: students are estimated as having a higher gaming tendency when they game in less commonly gamed contexts than when they only game in more commonly gamed contexts. IRT-GD outperformed the original detector on three datasets in terms of the association with learning. IRT-GD also more accurately revealed intervention effects on gaming and revealed a correlation between gaming and perceived competence in math. Our approach is not only useful for others wanting to apply a gaming assessment in their context but is also generally applicable in creating robust behavioral measures. [For the full proceedings, see ED623995.] |
| Abstractor: | As Provided |
| Entry Date: | 2022 |
| Accession Number: | ED624075 |
| Database: | ERIC |
| Abstract: | Gaming the system, a behavior in which learners exploit a system's properties to make progress while avoiding learning, has frequently been shown to be associated with lower learning. However, when we applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detected gaming was not associated with learning, challenging its construct validity. Our iterative exploratory data analysis suggested that some contextual factors that varied across and within conditions might contribute to this lack of association. We present a latent variable model, "item response theory-based gaming detection" (IRT-GD), that accounts for contextual factors and estimates latent gaming tendencies as the degree of deviation from normative behaviors across contexts. Item response theory models, widely used in knowledge assessment, account for item difficulty in estimating latent student abilities: students are estimated as having higher ability when they can get harder items correct than when they only get easier items correct. Similarly, IRT-GD accounts for contextual factors in estimating latent gaming tendencies: students are estimated as having a higher gaming tendency when they game in less commonly gamed contexts than when they only game in more commonly gamed contexts. IRT-GD outperformed the original detector on three datasets in terms of the association with learning. IRT-GD also more accurately revealed intervention effects on gaming and revealed a correlation between gaming and perceived competence in math. Our approach is not only useful for others wanting to apply a gaming assessment in their context but is also generally applicable in creating robust behavioral measures. [For the full proceedings, see ED623995.] |
|---|
