View in EDS

How Much Teacher Is in Teacher Rating Scales? EdWorkingPaper No. 23-828

Saved in:
Bibliographic Details
Title: How Much Teacher Is in Teacher Rating Scales? EdWorkingPaper No. 23-828
Language: English
Authors: D. Betsy McCoach (ORCID 0000-0001-9063-6835), Anthony J. Gambino (ORCID 0000-0001-5820-1997), Scott J. Peters (ORCID 0000-0003-2459-3384), Daniel Long (ORCID 0000-0003-1887-2993), Del Siegle (ORCID 0000-0001-5579-9217), Annenberg Institute for School Reform at Brown University
Source: Annenberg Institute for School Reform at Brown University. 2023.
Availability: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
Peer Reviewed: N
Page Count: 158
Publication Date: 2023
Document Type: Reports - Research
Descriptors: Rating Scales, Student Evaluation, Academically Gifted, Ability Identification, Bias, Effect Size, Cutting Scores, Evaluative Thinking, Reliability, Cognitive Measurement, Intelligence Tests, Children
Assessment and Survey Identifiers: Wechsler Intelligence Scale for Children
Abstract: Teacher rating scales (TRS) are often used to make service eligibility decisions for exceptional learners. Although TRS are regularly used to identify student exceptionalism either as part of an informal nomination process or through behavioral rating scales, there is little research documenting the between-teacher variance in teacher ratings or the consequences of such rater dependence. To evaluate the possible benefits or disadvantages of using TRS as part of a gifted identification process, we examined the student-, teacher-, and school-level variance in TRS controlling for student ability and achievement to determine the unique information, consistency, and potential bias in TRS. Between 10% and 25% of a students' TRS score can be attributed to the teacher doing the rating, and between-teacher standard deviations represent an effect size of one-third to one-half standard deviation unit. Our results suggest that TRS are not easily comparable across teachers, making it impossible to set a cut score for admission into a program (or for further screening) that functions equitably across teachers.
Abstractor: As Provided
Entry Date: 2025
Accession Number: ED672317
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED672317
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED672317
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: How Much Teacher Is in Teacher Rating Scales? EdWorkingPaper No. 23-828
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22D%2E+Betsy+McCoach%22">D. Betsy McCoach</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-9063-6835">0000-0001-9063-6835</externalLink>)<br /><searchLink fieldCode="AR" term="%22Anthony+J%2E+Gambino%22">Anthony J. Gambino</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-5820-1997">0000-0001-5820-1997</externalLink>)<br /><searchLink fieldCode="AR" term="%22Scott+J%2E+Peters%22">Scott J. Peters</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-2459-3384">0000-0003-2459-3384</externalLink>)<br /><searchLink fieldCode="AR" term="%22Daniel+Long%22">Daniel Long</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0003-1887-2993">0000-0003-1887-2993</externalLink>)<br /><searchLink fieldCode="AR" term="%22Del+Siegle%22">Del Siegle</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-5579-9217">0000-0001-5579-9217</externalLink>)<br /><searchLink fieldCode="AR" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22">Annenberg Institute for School Reform at Brown University</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Annenberg+Institute+for+School+Reform+at+Brown+University%22"><i>Annenberg Institute for School Reform at Brown University</i></searchLink>. 2023.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Annenberg Institute for School Reform at Brown University. Brown University Box 1985, Providence, RI 02912. Tel: 401-863-7990; Fax: 401-863-1290; e-mail: annenberg@brown.edu; Web site: https://annenberg.brown.edu/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: N
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 158
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2023
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Rating+Scales%22">Rating Scales</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Evaluation%22">Student Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Academically+Gifted%22">Academically Gifted</searchLink><br /><searchLink fieldCode="DE" term="%22Ability+Identification%22">Ability Identification</searchLink><br /><searchLink fieldCode="DE" term="%22Bias%22">Bias</searchLink><br /><searchLink fieldCode="DE" term="%22Effect+Size%22">Effect Size</searchLink><br /><searchLink fieldCode="DE" term="%22Cutting+Scores%22">Cutting Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Evaluative+Thinking%22">Evaluative Thinking</searchLink><br /><searchLink fieldCode="DE" term="%22Reliability%22">Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Cognitive+Measurement%22">Cognitive Measurement</searchLink><br /><searchLink fieldCode="DE" term="%22Intelligence+Tests%22">Intelligence Tests</searchLink><br /><searchLink fieldCode="DE" term="%22Children%22">Children</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Wechsler+Intelligence+Scale+for+Children%22">Wechsler Intelligence Scale for Children</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Teacher rating scales (TRS) are often used to make service eligibility decisions for exceptional learners. Although TRS are regularly used to identify student exceptionalism either as part of an informal nomination process or through behavioral rating scales, there is little research documenting the between-teacher variance in teacher ratings or the consequences of such rater dependence. To evaluate the possible benefits or disadvantages of using TRS as part of a gifted identification process, we examined the student-, teacher-, and school-level variance in TRS controlling for student ability and achievement to determine the unique information, consistency, and potential bias in TRS. Between 10% and 25% of a students' TRS score can be attributed to the teacher doing the rating, and between-teacher standard deviations represent an effect size of one-third to one-half standard deviation unit. Our results suggest that TRS are not easily comparable across teachers, making it impossible to set a cut score for admission into a program (or for further screening) that functions equitably across teachers.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED672317
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED672317
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 158
    Subjects:
      – SubjectFull: Rating Scales
        Type: general
      – SubjectFull: Student Evaluation
        Type: general
      – SubjectFull: Academically Gifted
        Type: general
      – SubjectFull: Ability Identification
        Type: general
      – SubjectFull: Bias
        Type: general
      – SubjectFull: Effect Size
        Type: general
      – SubjectFull: Cutting Scores
        Type: general
      – SubjectFull: Evaluative Thinking
        Type: general
      – SubjectFull: Reliability
        Type: general
      – SubjectFull: Cognitive Measurement
        Type: general
      – SubjectFull: Intelligence Tests
        Type: general
      – SubjectFull: Children
        Type: general
      – SubjectFull: Wechsler Intelligence Scale for Children
        Type: general
    Titles:
      – TitleFull: How Much Teacher Is in Teacher Rating Scales? EdWorkingPaper No. 23-828
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Annenberg Institute for School Reform at Brown University
      – PersonEntity:
          Name:
            NameFull: D. Betsy McCoach
      – PersonEntity:
          Name:
            NameFull: Anthony J. Gambino
      – PersonEntity:
          Name:
            NameFull: Scott J. Peters
      – PersonEntity:
          Name:
            NameFull: Daniel Long
      – PersonEntity:
          Name:
            NameFull: Del Siegle
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 08
              Type: published
              Y: 2023
          Titles:
            – TitleFull: Annenberg Institute for School Reform at Brown University
              Type: main
ResultId 1