Reliability Analysis of Test Questions to Support a National Tractor and Machinery Safety Certification Program
Saved in:
| Title: | Reliability Analysis of Test Questions to Support a National Tractor and Machinery Safety Certification Program |
|---|---|
| Language: | English |
| Authors: | Jill F. Kilanowski (ORCID |
| Source: | Journal of Agricultural Education. 2025 66(4). |
| Availability: | American Association for Agricultural Education. P.O. Box 7607, Department of Agricultural and Extension Education, Raleigh, NC 27695. Web site: https://jae-online.org/index.php/jae/index |
| Peer Reviewed: | Y |
| Page Count: | 17 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Education Level: | Junior High Schools Middle Schools Secondary Education High Schools |
| Descriptors: | Agricultural Education, Test Reliability, Agricultural Machinery, Safety Education, Certification, Student Certification, Computer Assisted Testing, Middle School Teachers, High School Teachers, Middle School Students, High School Students, Federal Legislation, Job Training, Job Skills |
| Geographic Terms: | Ohio, Montana, Kentucky |
| DOI: | 10.5032/jae.v66i4.3277 |
| ISSN: | 1042-0541 2162-5212 |
| Abstract: | The purpose of the study was to evaluate the reliability of a newly developed test bank of exam questions to support the U.S. Department of Labor tractor and machinery safety certificate's written exam. This test bank was developed by a panel of eight subject matter experts, from different U.S. geographic regions, in face-to-face meetings for item writing, to assess subject knowledge in middle and high school students aged 13-to-18 years old. The guiding theoretical frameworks were Webb's Depth of Knowledge and Bloom's Taxonomy. Via multiple recruitment efforts, high school agricultural educators and community 4-H advisors who taught youth certification programs were recruited to serve as classroom facilitators and replace their current knowledge exam with the study's WebXam online exam. Students (n = 96) from three states and six classrooms participated in the study, representing four different program formats. Results showed strong reliability (a = 0.93) for the exam, with no significant difference in the likelihood of passing based on age or sex. Seven themes were included in the 70-item test bank, where all questions were answered correctly by at least 30% of respondents, indicating there were no questions too difficult for the test-taker population. The results will serve as a foundation for establishing an online testing platform for educators who offer tractor and machinery safety trainings across the U.S., and in-turn may also create program consistency for the DOL written exam with a national test bank of valid and reliable questions. |
| Abstractor: | As Provided |
| Entry Date: | 2026 |
| Accession Number: | EJ1493665 |
| Database: | ERIC |
| Abstract: | The purpose of the study was to evaluate the reliability of a newly developed test bank of exam questions to support the U.S. Department of Labor tractor and machinery safety certificate's written exam. This test bank was developed by a panel of eight subject matter experts, from different U.S. geographic regions, in face-to-face meetings for item writing, to assess subject knowledge in middle and high school students aged 13-to-18 years old. The guiding theoretical frameworks were Webb's Depth of Knowledge and Bloom's Taxonomy. Via multiple recruitment efforts, high school agricultural educators and community 4-H advisors who taught youth certification programs were recruited to serve as classroom facilitators and replace their current knowledge exam with the study's WebXam online exam. Students (n = 96) from three states and six classrooms participated in the study, representing four different program formats. Results showed strong reliability (a = 0.93) for the exam, with no significant difference in the likelihood of passing based on age or sex. Seven themes were included in the 70-item test bank, where all questions were answered correctly by at least 30% of respondents, indicating there were no questions too difficult for the test-taker population. The results will serve as a foundation for establishing an online testing platform for educators who offer tractor and machinery safety trainings across the U.S., and in-turn may also create program consistency for the DOL written exam with a national test bank of valid and reliable questions. |
|---|---|
| ISSN: | 1042-0541 2162-5212 |
| DOI: | 10.5032/jae.v66i4.3277 |
