View in EDS

Reliability Analysis of Test Questions to Support a National Tractor and Machinery Safety Certification Program

Saved in:
Bibliographic Details
Title: Reliability Analysis of Test Questions to Support a National Tractor and Machinery Safety Certification Program
Language: English
Authors: Jill F. Kilanowski (ORCID 0000-0003-0455-1275), S. Dee Jepsen (ORCID 0000-0001-7221-5375), Rebecca Berenbon (ORCID 0000-0002-8744-3132)
Source: Journal of Agricultural Education. 2025 66(4).
Availability: American Association for Agricultural Education. P.O. Box 7607, Department of Agricultural and Extension Education, Raleigh, NC 27695. Web site: https://jae-online.org/index.php/jae/index
Peer Reviewed: Y
Page Count: 17
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Junior High Schools
Middle Schools
Secondary Education
High Schools
Descriptors: Agricultural Education, Test Reliability, Agricultural Machinery, Safety Education, Certification, Student Certification, Computer Assisted Testing, Middle School Teachers, High School Teachers, Middle School Students, High School Students, Federal Legislation, Job Training, Job Skills
Geographic Terms: Ohio, Montana, Kentucky
DOI: 10.5032/jae.v66i4.3277
ISSN: 1042-0541
2162-5212
Abstract: The purpose of the study was to evaluate the reliability of a newly developed test bank of exam questions to support the U.S. Department of Labor tractor and machinery safety certificate's written exam. This test bank was developed by a panel of eight subject matter experts, from different U.S. geographic regions, in face-to-face meetings for item writing, to assess subject knowledge in middle and high school students aged 13-to-18 years old. The guiding theoretical frameworks were Webb's Depth of Knowledge and Bloom's Taxonomy. Via multiple recruitment efforts, high school agricultural educators and community 4-H advisors who taught youth certification programs were recruited to serve as classroom facilitators and replace their current knowledge exam with the study's WebXam online exam. Students (n = 96) from three states and six classrooms participated in the study, representing four different program formats. Results showed strong reliability (a = 0.93) for the exam, with no significant difference in the likelihood of passing based on age or sex. Seven themes were included in the 70-item test bank, where all questions were answered correctly by at least 30% of respondents, indicating there were no questions too difficult for the test-taker population. The results will serve as a foundation for establishing an online testing platform for educators who offer tractor and machinery safety trainings across the U.S., and in-turn may also create program consistency for the DOL written exam with a national test bank of valid and reliable questions.
Abstractor: As Provided
Entry Date: 2026
Accession Number: EJ1493665
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1493665
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Reliability Analysis of Test Questions to Support a National Tractor and Machinery Safety Certification Program
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Jill+F%2E+Kilanowski%22">Jill F. Kilanowski</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-0455-1275">0000-0003-0455-1275</externalLink>)<br /><searchLink fieldCode="AR" term="%22S%2E+Dee+Jepsen%22">S. Dee Jepsen</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-7221-5375">0000-0001-7221-5375</externalLink>)<br /><searchLink fieldCode="AR" term="%22Rebecca+Berenbon%22">Rebecca Berenbon</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-8744-3132">0000-0002-8744-3132</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Agricultural+Education%22"><i>Journal of Agricultural Education</i></searchLink>. 2025 66(4).
– Name: Avail
  Label: Availability
  Group: Avail
  Data: American Association for Agricultural Education. P.O. Box 7607, Department of Agricultural and Extension Education, Raleigh, NC 27695. Web site: https://jae-online.org/index.php/jae/index
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 17
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink><br /><searchLink fieldCode="EL" term="%22High+Schools%22">High Schools</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Agricultural+Education%22">Agricultural Education</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reliability%22">Test Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Agricultural+Machinery%22">Agricultural Machinery</searchLink><br /><searchLink fieldCode="DE" term="%22Safety+Education%22">Safety Education</searchLink><br /><searchLink fieldCode="DE" term="%22Certification%22">Certification</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Certification%22">Student Certification</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Testing%22">Computer Assisted Testing</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Teachers%22">Middle School Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Teachers%22">High School Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Students%22">Middle School Students</searchLink><br /><searchLink fieldCode="DE" term="%22High+School+Students%22">High School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Federal+Legislation%22">Federal Legislation</searchLink><br /><searchLink fieldCode="DE" term="%22Job+Training%22">Job Training</searchLink><br /><searchLink fieldCode="DE" term="%22Job+Skills%22">Job Skills</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Ohio%22">Ohio</searchLink><br /><searchLink fieldCode="DE" term="%22Montana%22">Montana</searchLink><br /><searchLink fieldCode="DE" term="%22Kentucky%22">Kentucky</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.5032/jae.v66i4.3277
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1042-0541<br />2162-5212
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: The purpose of the study was to evaluate the reliability of a newly developed test bank of exam questions to support the U.S. Department of Labor tractor and machinery safety certificate's written exam. This test bank was developed by a panel of eight subject matter experts, from different U.S. geographic regions, in face-to-face meetings for item writing, to assess subject knowledge in middle and high school students aged 13-to-18 years old. The guiding theoretical frameworks were Webb's Depth of Knowledge and Bloom's Taxonomy. Via multiple recruitment efforts, high school agricultural educators and community 4-H advisors who taught youth certification programs were recruited to serve as classroom facilitators and replace their current knowledge exam with the study's WebXam online exam. Students (n = 96) from three states and six classrooms participated in the study, representing four different program formats. Results showed strong reliability (a = 0.93) for the exam, with no significant difference in the likelihood of passing based on age or sex. Seven themes were included in the 70-item test bank, where all questions were answered correctly by at least 30% of respondents, indicating there were no questions too difficult for the test-taker population. The results will serve as a foundation for establishing an online testing platform for educators who offer tractor and machinery safety trainings across the U.S., and in-turn may also create program consistency for the DOL written exam with a national test bank of valid and reliable questions.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1493665
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1493665
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.5032/jae.v66i4.3277
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 17
    Subjects:
      – SubjectFull: Agricultural Education
        Type: general
      – SubjectFull: Test Reliability
        Type: general
      – SubjectFull: Agricultural Machinery
        Type: general
      – SubjectFull: Safety Education
        Type: general
      – SubjectFull: Certification
        Type: general
      – SubjectFull: Student Certification
        Type: general
      – SubjectFull: Computer Assisted Testing
        Type: general
      – SubjectFull: Middle School Teachers
        Type: general
      – SubjectFull: High School Teachers
        Type: general
      – SubjectFull: Middle School Students
        Type: general
      – SubjectFull: High School Students
        Type: general
      – SubjectFull: Federal Legislation
        Type: general
      – SubjectFull: Job Training
        Type: general
      – SubjectFull: Job Skills
        Type: general
      – SubjectFull: Ohio
        Type: general
      – SubjectFull: Montana
        Type: general
      – SubjectFull: Kentucky
        Type: general
    Titles:
      – TitleFull: Reliability Analysis of Test Questions to Support a National Tractor and Machinery Safety Certification Program
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Jill F. Kilanowski
      – PersonEntity:
          Name:
            NameFull: S. Dee Jepsen
      – PersonEntity:
          Name:
            NameFull: Rebecca Berenbon
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-print
              Value: 1042-0541
            – Type: issn-electronic
              Value: 2162-5212
          Numbering:
            – Type: volume
              Value: 66
            – Type: issue
              Value: 4
          Titles:
            – TitleFull: Journal of Agricultural Education
              Type: main
ResultId 1