Assessing Graph Comprehension on Paper and Computer with MBA Students: A Crossover Experimental Study
Saved in:
| Title: | Assessing Graph Comprehension on Paper and Computer with MBA Students: A Crossover Experimental Study |
|---|---|
| Language: | English |
| Authors: | Boote, Stacy K., Boote, David N., Williamson, Steven |
| Source: | Cogent Education. 2021 8(1). |
| Availability: | Cogent OA. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals |
| Peer Reviewed: | Y |
| Page Count: | 21 |
| Publication Date: | 2021 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Higher Education Postsecondary Education |
| Descriptors: | Graduate Students, Masters Programs, Business Administration Education, Graphs, Computer Assisted Testing, Comprehension, Gender Differences, Test Format, Test Items, College Entrance Examinations, Business Schools, Graduate Study |
| Assessment and Survey Identifiers: | Graduate Management Admission Test |
| DOI: | 10.1080/2331186X.2021.1960247 |
| ISSN: | 2331-186X |
| Abstract: | Several decades of research suggesting differences in test performance across paper-based and computer-based assessments have been largely ameliorated through attention to test presentation equivalence, though no studies to date have focused on graph comprehension items. Test items requiring graph comprehension are increasingly common but may be especially prone to format effects. A crossover experimental design was used to compare paper-based (PBT) with computer-based test (CBT) formats in a sample of 28 MBA students answering 6 items requiring comprehension of Venn diagrams, scatterplots, and divided bar charts. Data analysis using a 2x2x2x2 ANOVA revealed that participants' better performance on CBT, [eta]2 = 0.07, was not statistically significant, p = 0.23. While DIF analysis suggested no difference by gender in total score or individual items, interaction analysis between gender and format showed the overall format effect was due mainly to males performing better on CBT, p = 0.02, d = 0.91. Females performed about the same in both formats. For scatterplot questions, participants also performed better on CBT, p < 0.005, [phi] = 0.41. Finally, participants were more likely to answer an easier question requiring attention to fewer graph components correctly after answering a more challenging question that required attention to all graph components, p = 0.02, [phi] = 0.5. Interaction analysis also revealed a large carryover effect from the research design ([eta]2 = 0.48, p = 0.000), which we interpreted as a learning effect. |
| Abstractor: | As Provided |
| Entry Date: | 2022 |
| Accession Number: | EJ1333783 |
| Database: | ERIC |
| Abstract: | Several decades of research suggesting differences in test performance across paper-based and computer-based assessments have been largely ameliorated through attention to test presentation equivalence, though no studies to date have focused on graph comprehension items. Test items requiring graph comprehension are increasingly common but may be especially prone to format effects. A crossover experimental design was used to compare paper-based (PBT) with computer-based test (CBT) formats in a sample of 28 MBA students answering 6 items requiring comprehension of Venn diagrams, scatterplots, and divided bar charts. Data analysis using a 2x2x2x2 ANOVA revealed that participants' better performance on CBT, [eta]2 = 0.07, was not statistically significant, p = 0.23. While DIF analysis suggested no difference by gender in total score or individual items, interaction analysis between gender and format showed the overall format effect was due mainly to males performing better on CBT, p = 0.02, d = 0.91. Females performed about the same in both formats. For scatterplot questions, participants also performed better on CBT, p < 0.005, [phi] = 0.41. Finally, participants were more likely to answer an easier question requiring attention to fewer graph components correctly after answering a more challenging question that required attention to all graph components, p = 0.02, [phi] = 0.5. Interaction analysis also revealed a large carryover effect from the research design ([eta]2 = 0.48, p = 0.000), which we interpreted as a learning effect. |
|---|---|
| ISSN: | 2331-186X |
| DOI: | 10.1080/2331186X.2021.1960247 |
