View in EDS

The Reliability and Consequential Validity of Two Teacher-Administered Student Mathematics Diagnostic Assessments. REL 2020-039

Saved in:
Bibliographic Details
Title: The Reliability and Consequential Validity of Two Teacher-Administered Student Mathematics Diagnostic Assessments. REL 2020-039
Language: English
Authors: Gersten, Russell, Jayanthi, Madhavi, Newman-Gonchar, Rebecca, Anderson, Daniel, Spallone, Samantha, Taylor, Mary Jo, National Center for Education Evaluation and Regional Assistance (ED), Regional Educational Laboratory Southeast (ED), Florida State University
Source: Regional Educational Laboratory Southeast. 2020.
Availability: Regional Educational Laboratory Southeast. Available from: Institute of Education Sciences. 550 12th Street SW, Washington, DC 20202. Tel: 202-245-6940; Web site: https://ies.ed.gov/ncee/edlabs/regions/southeast/index.asp
Peer Reviewed: Y
Page Count: 12
Publication Date: 2020
Contract Number: EDIES17C0011
Document Type: Reports - Research
Education Level: Early Childhood Education
Elementary Education
Grade 1
Primary Education
Grade 3
Descriptors: Mathematics Tests, Diagnostic Tests, Test Reliability, Test Validity, Interrater Reliability, Fidelity, Grade 1, Grade 3, Elementary School Teachers, Elementary School Students
Geographic Terms: Georgia
Abstract: Several school districts in Georgia use two teacher-administered diagnostic assessments of student knowledge of mathematics as part of their multi-tiered system of support in grades K-8: the Global Strategy Stage (GloSS; New Zealand Ministry of Education, 2012) and the Individual Knowledge Assessment of Number (IKAN; New Zealand Ministry of Education, 2011), which comes in two formats (Counting Interview and Written Assessment). However, little is known about whether two teachers using the same assessment to assess the same student on two occasions within a short period of time assign the same Stage Score (interassessor reliability) or about how useful the teachers found the assessments (consequential validity). Rather than relying on occasional testimonials from the field, decisions about using diagnostic assessments across the state should be based on psychometric data from an external source. Districts not currently using the GloSS and IKAN assessments have indicated an interest in using them, if they are proven to be reliable and valid diagnostic assessments, to assess students' understanding of mathematics and determine appropriate levels of instruction and intervention. This study found adequate interassessor reliability for the GloSS and for the IKAN Counting Interview but not for the IKAN Written Assessment. The IKAN Written Assessment requires additional attention to improve training so that reliability can be established. Teachers indicated that they found the screening data from the GloSS and IKAN assessments more useful for guiding decisions about student instruction and intervention than the screening data currently employed. Although teachers in the study's focus groups expressed strong support for both assessments, teachers reported in the study's survey that the GloSS is more useful than the IKAN because it assesses students' solution strategies, unlike most other mathematics assessments. Teachers also expressed some criticisms of both assessments; for example, they believed that the GloSS should include vocabulary familiar to students and that the IKAN Written Assessment should be untimed. [For the appendixes, see ED607586; for the study snapshot, see ED607587.]
Abstractor: As Provided
IES Funded: Yes
IES Publication: https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=4619
Entry Date: 2020
Accession Number: ED607585
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED607585
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED607585
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: The Reliability and Consequential Validity of Two Teacher-Administered Student Mathematics Diagnostic Assessments. REL 2020-039
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Gersten%2C+Russell%22">Gersten, Russell</searchLink><br /><searchLink fieldCode="AR" term="%22Jayanthi%2C+Madhavi%22">Jayanthi, Madhavi</searchLink><br /><searchLink fieldCode="AR" term="%22Newman-Gonchar%2C+Rebecca%22">Newman-Gonchar, Rebecca</searchLink><br /><searchLink fieldCode="AR" term="%22Anderson%2C+Daniel%22">Anderson, Daniel</searchLink><br /><searchLink fieldCode="AR" term="%22Spallone%2C+Samantha%22">Spallone, Samantha</searchLink><br /><searchLink fieldCode="AR" term="%22Taylor%2C+Mary+Jo%22">Taylor, Mary Jo</searchLink><br /><searchLink fieldCode="AR" term="%22National+Center+for+Education+Evaluation+and+Regional+Assistance+%28ED%29%22">National Center for Education Evaluation and Regional Assistance (ED)</searchLink><br /><searchLink fieldCode="AR" term="%22Regional+Educational+Laboratory+Southeast+%28ED%29%22">Regional Educational Laboratory Southeast (ED)</searchLink><br /><searchLink fieldCode="AR" term="%22Florida+State+University%22">Florida State University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Regional+Educational+Laboratory+Southeast%22"><i>Regional Educational Laboratory Southeast</i></searchLink>. 2020.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Regional Educational Laboratory Southeast. Available from: Institute of Education Sciences. 550 12th Street SW, Washington, DC 20202. Tel: 202-245-6940; Web site: https://ies.ed.gov/ncee/edlabs/regions/southeast/index.asp
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 12
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2020
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: EDIES17C0011
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Early+Childhood+Education%22">Early Childhood Education</searchLink><br /><searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+1%22">Grade 1</searchLink><br /><searchLink fieldCode="EL" term="%22Primary+Education%22">Primary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+3%22">Grade 3</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Mathematics+Tests%22">Mathematics Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Diagnostic+Tests%22">Diagnostic Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Reliability%22">Test Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Test+Validity%22">Test Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Fidelity%22">Fidelity</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+1%22">Grade 1</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+3%22">Grade 3</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+School+Teachers%22">Elementary School Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Elementary+School+Students%22">Elementary School Students</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Georgia%22">Georgia</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Several school districts in Georgia use two teacher-administered diagnostic assessments of student knowledge of mathematics as part of their multi-tiered system of support in grades K-8: the Global Strategy Stage (GloSS; New Zealand Ministry of Education, 2012) and the Individual Knowledge Assessment of Number (IKAN; New Zealand Ministry of Education, 2011), which comes in two formats (Counting Interview and Written Assessment). However, little is known about whether two teachers using the same assessment to assess the same student on two occasions within a short period of time assign the same Stage Score (interassessor reliability) or about how useful the teachers found the assessments (consequential validity). Rather than relying on occasional testimonials from the field, decisions about using diagnostic assessments across the state should be based on psychometric data from an external source. Districts not currently using the GloSS and IKAN assessments have indicated an interest in using them, if they are proven to be reliable and valid diagnostic assessments, to assess students' understanding of mathematics and determine appropriate levels of instruction and intervention. This study found adequate interassessor reliability for the GloSS and for the IKAN Counting Interview but not for the IKAN Written Assessment. The IKAN Written Assessment requires additional attention to improve training so that reliability can be established. Teachers indicated that they found the screening data from the GloSS and IKAN assessments more useful for guiding decisions about student instruction and intervention than the screening data currently employed. Although teachers in the study's focus groups expressed strong support for both assessments, teachers reported in the study's survey that the GloSS is more useful than the IKAN because it assesses students' solution strategies, unlike most other mathematics assessments. Teachers also expressed some criticisms of both assessments; for example, they believed that the GloSS should include vocabulary familiar to students and that the IKAN Written Assessment should be untimed. [For the appendixes, see ED607586; for the study snapshot, see ED607587.]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: CodeSource
  Label: IES Publication
  Group: SrcInfo
  Data: <link linkTarget="URL" linkTerm="https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=4619" linkWindow="_blank">https://ies.ed.gov/ncee/edlabs/projects/project.asp?projectID=4619</link>
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2020
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED607585
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED607585
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 12
    Subjects:
      – SubjectFull: Mathematics Tests
        Type: general
      – SubjectFull: Diagnostic Tests
        Type: general
      – SubjectFull: Test Reliability
        Type: general
      – SubjectFull: Test Validity
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: Fidelity
        Type: general
      – SubjectFull: Grade 1
        Type: general
      – SubjectFull: Grade 3
        Type: general
      – SubjectFull: Elementary School Teachers
        Type: general
      – SubjectFull: Elementary School Students
        Type: general
      – SubjectFull: Georgia
        Type: general
    Titles:
      – TitleFull: The Reliability and Consequential Validity of Two Teacher-Administered Student Mathematics Diagnostic Assessments. REL 2020-039
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: National Center for Education Evaluation and Regional Assistance (ED)
      – PersonEntity:
          Name:
            NameFull: Regional Educational Laboratory Southeast (ED)
      – PersonEntity:
          Name:
            NameFull: Florida State University
      – PersonEntity:
          Name:
            NameFull: Gersten, Russell
      – PersonEntity:
          Name:
            NameFull: Jayanthi, Madhavi
      – PersonEntity:
          Name:
            NameFull: Newman-Gonchar, Rebecca
      – PersonEntity:
          Name:
            NameFull: Anderson, Daniel
      – PersonEntity:
          Name:
            NameFull: Spallone, Samantha
      – PersonEntity:
          Name:
            NameFull: Taylor, Mary Jo
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 09
              Type: published
              Y: 2020
          Titles:
            – TitleFull: Regional Educational Laboratory Southeast
              Type: main
ResultId 1