Incorporating Test-Taking Engagement into Multistage Adaptive Testing Design for Large-Scale Assessments
Saved in:
| Title: | Incorporating Test-Taking Engagement into Multistage Adaptive Testing Design for Large-Scale Assessments |
|---|---|
| Language: | English |
| Authors: | Okan Bulut (ORCID |
| Source: | Journal of Educational Measurement. 2025 62(1):57-80. |
| Availability: | Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us |
| Peer Reviewed: | Y |
| Page Count: | 24 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Secondary Education |
| Descriptors: | Response Style (Tests), Testing Problems, Testing Accommodations, Measurement, Monte Carlo Methods, Learner Engagement, Reaction Time, Test Format, Achievement Tests, Foreign Countries, International Assessment, Sample Size |
| Assessment and Survey Identifiers: | Program for International Student Assessment |
| DOI: | 10.1111/jedm.12380 |
| ISSN: | 0022-0655 1745-3984 |
| Abstract: | The use of multistage adaptive testing (MST) has gradually increased in large-scale testing programs as MST achieves a balanced compromise between linear test design and item-level adaptive testing. MST works on the premise that each examinee gives their best effort when attempting the items, and their responses truly reflect what they know or can do. However, research shows that large-scale assessments may suffer from a lack of test-taking engagement, especially if they are low stakes. Examinees with low test-taking engagement are likely to show noneffortful responding (e.g., answering the items very rapidly without reading the item stem or response options). To alleviate the impact of noneffortful responses on the measurement accuracy of MST, test-taking engagement can be operationalized as a latent trait based on response times and incorporated into the on-the-fly module assembly procedure. To demonstrate the proposed approach, a Monte-Carlo simulation study was conducted based on item parameters from an international large-scale assessment. The results indicated that the on-the-fly module assembly considering both ability and test-taking engagement could minimize the impact of noneffortful responses, yielding more accurate ability estimates and classifications. Implications for practice and directions for future research were discussed. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1464057 |
| Database: | ERIC |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | The use of multistage adaptive testing (MST) has gradually increased in large-scale testing programs as MST achieves a balanced compromise between linear test design and item-level adaptive testing. MST works on the premise that each examinee gives their best effort when attempting the items, and their responses truly reflect what they know or can do. However, research shows that large-scale assessments may suffer from a lack of test-taking engagement, especially if they are low stakes. Examinees with low test-taking engagement are likely to show noneffortful responding (e.g., answering the items very rapidly without reading the item stem or response options). To alleviate the impact of noneffortful responses on the measurement accuracy of MST, test-taking engagement can be operationalized as a latent trait based on response times and incorporated into the on-the-fly module assembly procedure. To demonstrate the proposed approach, a Monte-Carlo simulation study was conducted based on item parameters from an international large-scale assessment. The results indicated that the on-the-fly module assembly considering both ability and test-taking engagement could minimize the impact of noneffortful responses, yielding more accurate ability estimates and classifications. Implications for practice and directions for future research were discussed. |
|---|---|
| ISSN: | 0022-0655 1745-3984 |
| DOI: | 10.1111/jedm.12380 |
