View in EDS

What's in a Score? A Decomposition of Observation Scores across Various Sources to Better Understand What Contributes to Scores

Saved in:
Bibliographic Details
Title: What's in a Score? A Decomposition of Observation Scores across Various Sources to Better Understand What Contributes to Scores
Language: English
Authors: Mark White
Source: Practical Assessment, Research & Evaluation. 2025 30.
Availability: University of Massachusetts Amherst Libraries. 154 Hicks Way, Amherst, MA 01003. e-mail: pare@umass.edu; Web site: https://openpublishing.library.umass.edu/pare/
Peer Reviewed: Y
Page Count: 34
Publication Date: 2025
Document Type: Journal Articles
Reports - Research
Education Level: Junior High Schools
Middle Schools
Secondary Education
Descriptors: Scores, Teacher Effectiveness, Teacher Evaluation, Observation, Accuracy, Interrater Reliability, Teaching Conditions, Student Characteristics, Teacher Characteristics, English Teachers, Middle School Teachers, Middle School Students, Classroom Environment
Assessment and Survey Identifiers: Classroom Assessment Scoring System
ISSN: 1531-7714
Abstract: Systematized, observational approaches to measuring teaching quality are an important tool in research and practice. Termed observation systems, these approaches include a rubric that operationalizes a set of teaching quality constructs and structures to support rater training and monitoring. Scores from observation systems, through their interpretation as capturing the intended teaching quality constructs, are used to develop theoretical understandings of teaching quality. This paper explores what factors contribute to observation scores in a secondary analysis of the Understanding Teaching Quality project. Leveraging calibration data, I combine mixed-effects regression analyses of calibration data that examine rater accuracy (i.e., deviations from master scores) with analyses of operational data to explore the extent to which raters, students, teachers, and the teaching context contribute to scores. These analyses highlight that (1) some rater error may be invisible in typical analyses examining rater agreement; (2) rater error is largely systematic; and (3) differences in student composition across schools largely explain between-school differences in scores. These results highlight potential biases in estimates of score reliability and validity coefficients that might exist in studies that model rater agreement rather than rater accuracy and/or that fail to consider differences in between-teacher and between-school variation in observation scores.
Abstractor: As Provided
Entry Date: 2025
Accession Number: EJ1472538
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1472538
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1472538
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: What's in a Score? A Decomposition of Observation Scores across Various Sources to Better Understand What Contributes to Scores
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Mark+White%22">Mark White</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Practical+Assessment%2C+Research+%26+Evaluation%22"><i>Practical Assessment, Research & Evaluation</i></searchLink>. 2025 30.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: University of Massachusetts Amherst Libraries. 154 Hicks Way, Amherst, MA 01003. e-mail: pare@umass.edu; Web site: https://openpublishing.library.umass.edu/pare/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 34
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Scores%22">Scores</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Effectiveness%22">Teacher Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Evaluation%22">Teacher Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Observation%22">Observation</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Conditions%22">Teaching Conditions</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Characteristics%22">Student Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Characteristics%22">Teacher Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22English+Teachers%22">English Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Teachers%22">Middle School Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Students%22">Middle School Students</searchLink><br /><searchLink fieldCode="DE" term="%22Classroom+Environment%22">Classroom Environment</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Classroom+Assessment+Scoring+System%22">Classroom Assessment Scoring System</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 1531-7714
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Systematized, observational approaches to measuring teaching quality are an important tool in research and practice. Termed observation systems, these approaches include a rubric that operationalizes a set of teaching quality constructs and structures to support rater training and monitoring. Scores from observation systems, through their interpretation as capturing the intended teaching quality constructs, are used to develop theoretical understandings of teaching quality. This paper explores what factors contribute to observation scores in a secondary analysis of the Understanding Teaching Quality project. Leveraging calibration data, I combine mixed-effects regression analyses of calibration data that examine rater accuracy (i.e., deviations from master scores) with analyses of operational data to explore the extent to which raters, students, teachers, and the teaching context contribute to scores. These analyses highlight that (1) some rater error may be invisible in typical analyses examining rater agreement; (2) rater error is largely systematic; and (3) differences in student composition across schools largely explain between-school differences in scores. These results highlight potential biases in estimates of score reliability and validity coefficients that might exist in studies that model rater agreement rather than rater accuracy and/or that fail to consider differences in between-teacher and between-school variation in observation scores.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1472538
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1472538
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 34
    Subjects:
      – SubjectFull: Scores
        Type: general
      – SubjectFull: Teacher Effectiveness
        Type: general
      – SubjectFull: Teacher Evaluation
        Type: general
      – SubjectFull: Observation
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: Teaching Conditions
        Type: general
      – SubjectFull: Student Characteristics
        Type: general
      – SubjectFull: Teacher Characteristics
        Type: general
      – SubjectFull: English Teachers
        Type: general
      – SubjectFull: Middle School Teachers
        Type: general
      – SubjectFull: Middle School Students
        Type: general
      – SubjectFull: Classroom Environment
        Type: general
      – SubjectFull: Classroom Assessment Scoring System
        Type: general
    Titles:
      – TitleFull: What's in a Score? A Decomposition of Observation Scores across Various Sources to Better Understand What Contributes to Scores
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Mark White
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2025
          Identifiers:
            – Type: issn-electronic
              Value: 1531-7714
          Numbering:
            – Type: volume
              Value: 30
          Titles:
            – TitleFull: Practical Assessment, Research & Evaluation
              Type: main
ResultId 1