View in EDS

Opportunities and Challenges in the Use of Video in Early Childhood Classroom Quality Observations

Saved in:
Bibliographic Details
Title: Opportunities and Challenges in the Use of Video in Early Childhood Classroom Quality Observations
Language: English
Authors: Anne Partika, Sarah Gerard, Todd Grindal, Gullnar Sy, Morgan Solender, Society for Research on Educational Effectiveness (SREE)
Source: Society for Research on Educational Effectiveness. 2024.
Availability: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Peer Reviewed: Y
Publication Date: 2024
Document Type: Reports - Research
Education Level: Early Childhood Education
Preschool Education
Descriptors: Video Technology, Classroom Observation Techniques, Barriers, Faculty Development, Coaching (Performance), Accountability, Early Childhood Education, Accuracy, Usability, Costs, Professional Autonomy, Teacher Characteristics, Minority Group Teachers, Preschool Teachers, Classroom Environment, Educational Quality, Rating Scales, Interrater Reliability, Teacher Surveys, Teacher Attitudes, Administrator Attitudes, Charter Schools, Religious Schools, Public Schools, Child Care Centers, Preschools
Geographic Terms: Maryland, Virginia, District of Columbia, Massachusetts
Assessment and Survey Identifiers: Classroom Assessment Scoring System, Early Childhood Environment Rating Scale
Abstract: Background & Context: Accurate, reliable, and scalable measurement of classroom quality is critical to ensure that children benefit from early childhood programs. Early childhood classroom quality is most commonly measured using classroom observation tools serving multiple purposes including (1) teacher support (e.g., coaching, professional development), (2) accountability systems (e.g., quality rating and improvement systems), and (3) education research. Education policymakers have expressed interest in using video technology to improve the accuracy, usefulness, and reduce the costs of classroom observations (Curby et al., 2016; Kane et al., 2020). The use of video recordings might provide classroom teachers with greater agency in the timing of observations and opportunities to review the results (Teachstone & Intentional Futures, 2023). Video might also enable researchers to better understand how early educational experiences differ for racially, culturally, and linguistically minoritized learners (Meek et al., 2022). Although some prior research has documented the comparability of classroom observations in homogenous contexts (Curby et al., 2016), little is known about the reliability of video-based observations in the wider range of publicly funded preschool classrooms, nor the extent to which they may feasibly be implemented to support teacher coaching, improve accountability systems, and provide data on classrooms. Objective: We address this gap by examining the reliability of scores from video based and live observations of early childhood classrooms. We do this with two observation tools commonly used in early childhood settings: the Classroom Assessment Scoring System PreK-3rd (CLASS 2nd edition; Pianta & Hamre, 2022) and the Early Childhood Environment Rating Scales (ECERS-3; Harms et al., 2014). We address three research questions: (1) Do live and video observations, respectively, have sufficient inter-rater reliability (IRR) within-condition?; (2) Do live and video observations have sufficient between-condition IRR (i.e., do observers assign similar ratings to the same observation period scored on video and live)?; and (3) Do scores on domains/subscales and dimensions/items vary systematically across live and video-based coders? Additionally, we will address exploratory questions about the feasibility of video-based classroom observations using data from teacher surveys and interviews with teachers and program leaders. Setting: The study team conducted observations in preschool programs in Washington DC, Maryland, Massachusetts, and Virginia, including public pre-K, public charter schools, community-based childcare centers, faith-based programs, and cooperative nursery schools. Participants: The sample includes 100 CLASS and 60 ECERS observations. Most participating lead teachers (88%) identified as women; 6% identify as Asian, 26% identify as Black, 16% identify as Hispanic/Latine, 42% identify as White, and 6% identify as multiracial. Most teachers (77%) had at least a B.A. Years of teaching experience ranged from 1 to 45 years. Practice: This study focuses on identifying whether the practice of conducting classroom observations via video is reliable, feasible, and useful to support teacher professional development, accountability systems, and research. We include a tool focused on process quality (CLASS) and a tool that includes both process and structural quality (ECERS-3). Research Design: This mixed-methods study (preregistered at https://osf.io/hy638) includes psychometric analyses of reliability and differential item functioning, descriptive analysis of teacher survey data, and thematic analysis using inductive coding methods to synthesize findings across qualitative teacher and program leader interviews. Data Collection and Analysis: Certified CLASS 2nd edition and ECERS-3 observers conducted observations live while simultaneously recording the classroom to be coded later by a different certified observer. Videos were recorded using a stationary recording device that rotated to track the lead teacher's movements and microphones worn by teachers and observers. Following the observation, all lead teachers were invited to complete a survey about the observation experience and the usefulness, burden, and perceived accuracy of observations. Selected teachers and program directors were also invited to participate in interviews. To understand whether live and video observations produce reliable and equivalent ratings, we first calculated within-condition IRR separately for live and video observations. Second, we assessed IRR between conditions by calculating agreement where the live and video observations, respectively, are considered different raters. Third, we used multivariate regression models to predict scores from condition, controlling for observer to account for the fact that observers were not randomly distributed across live and video observations. To provide insights on exploratory questions about feasibility, we will incorporate descriptive statistics from teacher surveys and use inductive coding methods to identify themes from the teacher and program leader interviews. Preliminary Results: Preliminary results indicate that video-based scoring of the CLASS has sufficient within-condition IRR (88%, above the standard cut-off of 80%; Landis & Koch, 1977). Additionally, observers have sufficient between-condition IRR, as preliminary results indicate 89% percent-within-one agreement between live and video raters of the same observation. Yet despite these levels of reliability, scores reflected somewhat higher quality teaching for live observations than video across several dimensions of the CLASS (Figure 1). As such, preliminary evidence indicates that despite meeting reliability standards, using video observations interchangeably with live scores or in comparison to quality cut-offs based on live scores may disadvantage classrooms observed via video. Next steps for analysis include (1) re-estimating analyses on the final sample; (2) replicating the analyses for the CLASS with the ECERS; and (3) conducting descriptive and thematic analysis of survey and interview data. Conclusions: Understanding the reliability, feasibility, and utility of video-based classroom observations is essential as the educational landscape integrates increasingly advanced technology into professional development, accountability, and research. Preliminary findings highlight how evidence of the effectiveness of new technologies in education should be considered with insights from practitioners and in light of the distinct use case scenarios. For example, results suggest video-based CLASS observations have sufficient reliability and as such, may be useful for instructional coaching. However, the systematic differences in scores across live and video observations suggests that, if used in high-stakes accountability settings such as QRIS, programs or teachers scored on video may be disadvantaged compared to those scored live. While the results of this study focus on current use cases, results have implications for scenarios that education researchers and practitioners may encounter in the future.
Abstractor: As Provided
Entry Date: 2025
Access URL: https://www.sree.org/2024-conference
Accession Number: ED663360
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: ED663360
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Opportunities and Challenges in the Use of Video in Early Childhood Classroom Quality Observations
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Anne+Partika%22">Anne Partika</searchLink><br /><searchLink fieldCode="AR" term="%22Sarah+Gerard%22">Sarah Gerard</searchLink><br /><searchLink fieldCode="AR" term="%22Todd+Grindal%22">Todd Grindal</searchLink><br /><searchLink fieldCode="AR" term="%22Gullnar+Sy%22">Gullnar Sy</searchLink><br /><searchLink fieldCode="AR" term="%22Morgan+Solender%22">Morgan Solender</searchLink><br /><searchLink fieldCode="AR" term="%22Society+for+Research+on+Educational+Effectiveness+%28SREE%29%22">Society for Research on Educational Effectiveness (SREE)</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Society+for+Research+on+Educational+Effectiveness%22"><i>Society for Research on Educational Effectiveness</i></searchLink>. 2024.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Early+Childhood+Education%22">Early Childhood Education</searchLink><br /><searchLink fieldCode="EL" term="%22Preschool+Education%22">Preschool Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Video+Technology%22">Video Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Classroom+Observation+Techniques%22">Classroom Observation Techniques</searchLink><br /><searchLink fieldCode="DE" term="%22Barriers%22">Barriers</searchLink><br /><searchLink fieldCode="DE" term="%22Faculty+Development%22">Faculty Development</searchLink><br /><searchLink fieldCode="DE" term="%22Coaching+%28Performance%29%22">Coaching (Performance)</searchLink><br /><searchLink fieldCode="DE" term="%22Accountability%22">Accountability</searchLink><br /><searchLink fieldCode="DE" term="%22Early+Childhood+Education%22">Early Childhood Education</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Usability%22">Usability</searchLink><br /><searchLink fieldCode="DE" term="%22Costs%22">Costs</searchLink><br /><searchLink fieldCode="DE" term="%22Professional+Autonomy%22">Professional Autonomy</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Characteristics%22">Teacher Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Minority+Group+Teachers%22">Minority Group Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Preschool+Teachers%22">Preschool Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Classroom+Environment%22">Classroom Environment</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Quality%22">Educational Quality</searchLink><br /><searchLink fieldCode="DE" term="%22Rating+Scales%22">Rating Scales</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Surveys%22">Teacher Surveys</searchLink><br /><searchLink fieldCode="DE" term="%22Teacher+Attitudes%22">Teacher Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Administrator+Attitudes%22">Administrator Attitudes</searchLink><br /><searchLink fieldCode="DE" term="%22Charter+Schools%22">Charter Schools</searchLink><br /><searchLink fieldCode="DE" term="%22Religious+Schools%22">Religious Schools</searchLink><br /><searchLink fieldCode="DE" term="%22Public+Schools%22">Public Schools</searchLink><br /><searchLink fieldCode="DE" term="%22Child+Care+Centers%22">Child Care Centers</searchLink><br /><searchLink fieldCode="DE" term="%22Preschools%22">Preschools</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Maryland%22">Maryland</searchLink><br /><searchLink fieldCode="DE" term="%22Virginia%22">Virginia</searchLink><br /><searchLink fieldCode="DE" term="%22District+of+Columbia%22">District of Columbia</searchLink><br /><searchLink fieldCode="DE" term="%22Massachusetts%22">Massachusetts</searchLink>
– Name: SubjectThesaurus
  Label: Assessment and Survey Identifiers
  Group: Su
  Data: <searchLink fieldCode="SU" term="%22Classroom+Assessment+Scoring+System%22">Classroom Assessment Scoring System</searchLink><br /><searchLink fieldCode="SU" term="%22Early+Childhood+Environment+Rating+Scale%22">Early Childhood Environment Rating Scale</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background & Context: Accurate, reliable, and scalable measurement of classroom quality is critical to ensure that children benefit from early childhood programs. Early childhood classroom quality is most commonly measured using classroom observation tools serving multiple purposes including (1) teacher support (e.g., coaching, professional development), (2) accountability systems (e.g., quality rating and improvement systems), and (3) education research. Education policymakers have expressed interest in using video technology to improve the accuracy, usefulness, and reduce the costs of classroom observations (Curby et al., 2016; Kane et al., 2020). The use of video recordings might provide classroom teachers with greater agency in the timing of observations and opportunities to review the results (Teachstone & Intentional Futures, 2023). Video might also enable researchers to better understand how early educational experiences differ for racially, culturally, and linguistically minoritized learners (Meek et al., 2022). Although some prior research has documented the comparability of classroom observations in homogenous contexts (Curby et al., 2016), little is known about the reliability of video-based observations in the wider range of publicly funded preschool classrooms, nor the extent to which they may feasibly be implemented to support teacher coaching, improve accountability systems, and provide data on classrooms. Objective: We address this gap by examining the reliability of scores from video based and live observations of early childhood classrooms. We do this with two observation tools commonly used in early childhood settings: the Classroom Assessment Scoring System PreK-3rd (CLASS 2nd edition; Pianta & Hamre, 2022) and the Early Childhood Environment Rating Scales (ECERS-3; Harms et al., 2014). We address three research questions: (1) Do live and video observations, respectively, have sufficient inter-rater reliability (IRR) within-condition?; (2) Do live and video observations have sufficient between-condition IRR (i.e., do observers assign similar ratings to the same observation period scored on video and live)?; and (3) Do scores on domains/subscales and dimensions/items vary systematically across live and video-based coders? Additionally, we will address exploratory questions about the feasibility of video-based classroom observations using data from teacher surveys and interviews with teachers and program leaders. Setting: The study team conducted observations in preschool programs in Washington DC, Maryland, Massachusetts, and Virginia, including public pre-K, public charter schools, community-based childcare centers, faith-based programs, and cooperative nursery schools. Participants: The sample includes 100 CLASS and 60 ECERS observations. Most participating lead teachers (88%) identified as women; 6% identify as Asian, 26% identify as Black, 16% identify as Hispanic/Latine, 42% identify as White, and 6% identify as multiracial. Most teachers (77%) had at least a B.A. Years of teaching experience ranged from 1 to 45 years. Practice: This study focuses on identifying whether the practice of conducting classroom observations via video is reliable, feasible, and useful to support teacher professional development, accountability systems, and research. We include a tool focused on process quality (CLASS) and a tool that includes both process and structural quality (ECERS-3). Research Design: This mixed-methods study (preregistered at https://osf.io/hy638) includes psychometric analyses of reliability and differential item functioning, descriptive analysis of teacher survey data, and thematic analysis using inductive coding methods to synthesize findings across qualitative teacher and program leader interviews. Data Collection and Analysis: Certified CLASS 2nd edition and ECERS-3 observers conducted observations live while simultaneously recording the classroom to be coded later by a different certified observer. Videos were recorded using a stationary recording device that rotated to track the lead teacher's movements and microphones worn by teachers and observers. Following the observation, all lead teachers were invited to complete a survey about the observation experience and the usefulness, burden, and perceived accuracy of observations. Selected teachers and program directors were also invited to participate in interviews. To understand whether live and video observations produce reliable and equivalent ratings, we first calculated within-condition IRR separately for live and video observations. Second, we assessed IRR between conditions by calculating agreement where the live and video observations, respectively, are considered different raters. Third, we used multivariate regression models to predict scores from condition, controlling for observer to account for the fact that observers were not randomly distributed across live and video observations. To provide insights on exploratory questions about feasibility, we will incorporate descriptive statistics from teacher surveys and use inductive coding methods to identify themes from the teacher and program leader interviews. Preliminary Results: Preliminary results indicate that video-based scoring of the CLASS has sufficient within-condition IRR (88%, above the standard cut-off of 80%; Landis & Koch, 1977). Additionally, observers have sufficient between-condition IRR, as preliminary results indicate 89% percent-within-one agreement between live and video raters of the same observation. Yet despite these levels of reliability, scores reflected somewhat higher quality teaching for live observations than video across several dimensions of the CLASS (Figure 1). As such, preliminary evidence indicates that despite meeting reliability standards, using video observations interchangeably with live scores or in comparison to quality cut-offs based on live scores may disadvantage classrooms observed via video. Next steps for analysis include (1) re-estimating analyses on the final sample; (2) replicating the analyses for the CLASS with the ECERS; and (3) conducting descriptive and thematic analysis of survey and interview data. Conclusions: Understanding the reliability, feasibility, and utility of video-based classroom observations is essential as the educational landscape integrates increasingly advanced technology into professional development, accountability, and research. Preliminary findings highlight how evidence of the effectiveness of new technologies in education should be considered with insights from practitioners and in light of the distinct use case scenarios. For example, results suggest video-based CLASS observations have sufficient reliability and as such, may be useful for instructional coaching. However, the systematic differences in scores across live and video observations suggests that, if used in high-stakes accountability settings such as QRIS, programs or teachers scored on video may be disadvantaged compared to those scored live. While the results of this study focus on current use cases, results have implications for scenarios that education researchers and practitioners may encounter in the future.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2025
– Name: URL
  Label: Access URL
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://www.sree.org/2024-conference" linkWindow="_blank">https://www.sree.org/2024-conference</link>
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED663360
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED663360
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    Subjects:
      – SubjectFull: Video Technology
        Type: general
      – SubjectFull: Classroom Observation Techniques
        Type: general
      – SubjectFull: Barriers
        Type: general
      – SubjectFull: Faculty Development
        Type: general
      – SubjectFull: Coaching (Performance)
        Type: general
      – SubjectFull: Accountability
        Type: general
      – SubjectFull: Early Childhood Education
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Usability
        Type: general
      – SubjectFull: Costs
        Type: general
      – SubjectFull: Professional Autonomy
        Type: general
      – SubjectFull: Teacher Characteristics
        Type: general
      – SubjectFull: Minority Group Teachers
        Type: general
      – SubjectFull: Preschool Teachers
        Type: general
      – SubjectFull: Classroom Environment
        Type: general
      – SubjectFull: Educational Quality
        Type: general
      – SubjectFull: Rating Scales
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
      – SubjectFull: Teacher Surveys
        Type: general
      – SubjectFull: Teacher Attitudes
        Type: general
      – SubjectFull: Administrator Attitudes
        Type: general
      – SubjectFull: Charter Schools
        Type: general
      – SubjectFull: Religious Schools
        Type: general
      – SubjectFull: Public Schools
        Type: general
      – SubjectFull: Child Care Centers
        Type: general
      – SubjectFull: Preschools
        Type: general
      – SubjectFull: Maryland
        Type: general
      – SubjectFull: Virginia
        Type: general
      – SubjectFull: District of Columbia
        Type: general
      – SubjectFull: Massachusetts
        Type: general
      – SubjectFull: Classroom Assessment Scoring System
        Type: general
      – SubjectFull: Early Childhood Environment Rating Scale
        Type: general
    Titles:
      – TitleFull: Opportunities and Challenges in the Use of Video in Early Childhood Classroom Quality Observations
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Society for Research on Educational Effectiveness (SREE)
      – PersonEntity:
          Name:
            NameFull: Anne Partika
      – PersonEntity:
          Name:
            NameFull: Sarah Gerard
      – PersonEntity:
          Name:
            NameFull: Todd Grindal
      – PersonEntity:
          Name:
            NameFull: Gullnar Sy
      – PersonEntity:
          Name:
            NameFull: Morgan Solender
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 19
              M: 09
              Type: published
              Y: 2024
          Titles:
            – TitleFull: Society for Research on Educational Effectiveness
              Type: main
ResultId 1