View in EDS

Understanding Disruptions to Virtual Learning: Causes of and Variation in Lost Instructional Time

Saved in:
Bibliographic Details
Title: Understanding Disruptions to Virtual Learning: Causes of and Variation in Lost Instructional Time
Language: English
Authors: Xander Beberman, Sarah Novicoff, Ana Trindade Ribeiro, Carly Robinson, Susanna Loeb, Society for Research on Educational Effectiveness (SREE)
Source: Society for Research on Educational Effectiveness. 2025.
Availability: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Peer Reviewed: Y
Publication Date: 2025
Document Type: Reports - Research
Education Level: Early Childhood Education
Elementary Education
Kindergarten
Primary Education
Grade 1
Grade 2
Descriptors: Electronic Learning, Tutoring, Student Characteristics, Institutional Characteristics, Public Schools, Kindergarten, Grade 1, Grade 2, Program Effectiveness, Emergent Literacy, Literacy Education, Time Factors (Learning), Classification
Geographic Terms: Texas
Abstract: Background/Context: Virtual tutoring has grown significantly as a personalized learning tool, especially in response to the academic disruptions caused by the COVID-19 pandemic. While out-of-school virtual tutoring has demonstrated moderate to large positive effects on student learning (0.20-0.40 SDs; Carlana and La Ferrara, 2024; Deacon and Chojnacki, 2023; Gortazar et al., 2024; Roschelle et al., 2020), in-school virtual tutoring shows significantly smaller impacts (0.01-0.08 SDs; Kraft et al., 2022; Ready et al., 2024; Robinson et al., 2024). Although many factors may explain this discrepancy, we hypothesize that in-school virtual tutoring can be more susceptible to disruptions similar to classroom instruction (Kraft and Monti-Nussbaum, 2021), affecting the overall instruction time. Prior research has shown that instructional time is a critical determinant of learning outcomes (Yesil Dagli (2019); Gromada and Shewbridge (2016); Holland et al. (2015); Kraft and Novicoff (2024); Patall et al. (2010), but due to difficulties in data collection, there has been limited empirical work quantifying how disruptions can reduce instruction time and affect learning. Purpose/Objective/Research Question: The primary objective of this study is to investigate how much instructional time is lost due to disruptions during in-school virtual tutoring sessions and to analyze the relationship between these disruptions and student and school characteristics. Specifically, the study aims to identify different types of disruptions and quantify the time they consume during tutoring, using a text classification model trained on tutor transcripts annotated by humans. The underlying hypothesis is that a substantial portion of virtual tutoring time in schools is spent addressing disruptions, which contributes to the diminished effectiveness observed in such settings. Setting: Our study uses data collected for the RCT conducted in Robinson et al., 2024. The virtual tutoring intervention occurred across 12 campuses within the same public school system in Texas. Tutoring sessions were administered virtually during the school day, using a platform that connected students with remote tutors nationwide to deliver early literacy instruction to participating students assigned to one of the treatment groups. Population/Participants/Subjects: Data was collected from 1,357 students in kindergarten through second grade and 192 tutors who participated in an RCT to evaluate the effectiveness of in-school virtual tutoring (Robinson et al., 2024). Over 26,000 tutoring sessions were analyzed, with 19,448 individual tutoring sessions (1:1 student-tutor ratio) and 7,150 paired tutoring sessions (2:1 tutor-student ratio). Students were diverse in terms of demographics and learning needs, including those receiving free or reduced lunch, English learners, and students with disabilities. Tutoring was focused on early literacy development, and the main outcomes for the RCT were achievement scores on DIBELS and MAP Reading. Intervention/Program/Practice: The tutoring intervention involved early literacy virtual instruction delivered via a video conferencing platform. Tutors and students engaged in individualized or paired tutoring sessions using structured literacy materials. Sessions were recorded, and transcript and audio data were collected. A typical session lasted around 20-30 minutes, and the program implementation occurred from November 2022 to May 2023. Research Design: We use natural language processing (NLP) to classify and analyze types of disruptions during virtual tutoring. We developed a custom multi-class text classifier trained on tutor utterances labeled by human annotators. Sessions were selected from a two-week window in April 2023, and regression models were used to assess the relationship between session characteristics related to the students and their school and the prevalence of different types of disruptions. Data Collection and Analysis: We transcribed all 26,714 session recordings from the RCT's tutoring sessions for this study. We created a human-annotated set of 2,501 annotations of 1,488 unique utterances from 255 sessions to train our classifier. We achieved an inter-rater reliability of 0.71 using Krippendorff's alpha across 19 human annotators. Disruption categories included tech problems, student disruptions, background noise, and issues involving additional students or substitute tutors. A RoBERTa-based classifier was fine-tuned using 80% of the labeled data, with oversampling of less frequent categories to address class imbalance. Accuracy reached 74% overall (80% for disruption vs. non-disruption). We estimated disruption durations based on the classifier labeling of each utterance and utterance timestamps. Our analysis used session-level aggregation of disruption time, examining how disruption time varied by student composition (e.g., 1:1 vs. 2:1 tutoring), demographics, and school for the students in the session. We estimate the following regression, where i is the session, y_i is the percentage of session time spent on one of the categories from our disruption classifier, 2:1 is a dummy for students assigned to paired tutoring (instead of individual), X represents a vector of student demographics and other characteristics, [delta] represents school (s) fixed effects, and represents grade fixed effects. y[subscript i] = [beta subscript 0] + [beta subscript 1] (2 : 1t)[subscript i] + X[subscript i][gamma] + [delta subscript s(i)] + [lambda subscript g(i)] + [epsilon subscript i]. Findings/Results: We found that approximately 25% of tutoring time was spent managing disruptions. Of this, 11.8% of time was attributed to student behavior, and 9% to technology issues. Pair tutoring (2:1) sessions were associated with significantly less usable instructional time--nearly 9 percentage points lower than individual (1:1) sessions. Sessions with at least one female student had more usable instructional time (+3 p.p.), suggesting possible behavioral differences by gender. Tech issues were less prevalent among second-grade students, likely due to their greater experience with technology. We also found that tech issues and background noise disruptions correlated more strongly with specific schools, suggesting variations in the implementation environment affected the amount of instruction time available to students through virtual tutoring. Conclusions: Disruptions are a significant drain on instructional time in in-school virtual tutoring settings, with both student behavior and environmental factors contributing. Notably, disruptions varied systematically by session type, student demographics, and school, suggesting that targeted program design and implementation changes--such as quieter tutoring locations, better tech support, and smaller group sizes--could mitigate some of these effects. As we continue to work on this study, our next steps include extending our analysis to the full set of tutoring sessions and investigating whether lower levels of school environment disruptions, such as tech issues and background noise, also correlate with larger learning gains for students by estimating differential impact on student performance on the end-of-year DIBELS assessment.
Abstractor: As Provided
Entry Date: 2026
Access URL: https://www.sree.org/2025-conference
Accession Number: ED677776
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: ED677776
AccessLevel: 3
PubType: Report
PubTypeId: report
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Understanding Disruptions to Virtual Learning: Causes of and Variation in Lost Instructional Time
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Xander+Beberman%22">Xander Beberman</searchLink><br /><searchLink fieldCode="AR" term="%22Sarah+Novicoff%22">Sarah Novicoff</searchLink><br /><searchLink fieldCode="AR" term="%22Ana+Trindade+Ribeiro%22">Ana Trindade Ribeiro</searchLink><br /><searchLink fieldCode="AR" term="%22Carly+Robinson%22">Carly Robinson</searchLink><br /><searchLink fieldCode="AR" term="%22Susanna+Loeb%22">Susanna Loeb</searchLink><br /><searchLink fieldCode="AR" term="%22Society+for+Research+on+Educational+Effectiveness+%28SREE%29%22">Society for Research on Educational Effectiveness (SREE)</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Society+for+Research+on+Educational+Effectiveness%22"><i>Society for Research on Educational Effectiveness</i></searchLink>. 2025.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2025
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Early+Childhood+Education%22">Early Childhood Education</searchLink><br /><searchLink fieldCode="EL" term="%22Elementary+Education%22">Elementary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Kindergarten%22">Kindergarten</searchLink><br /><searchLink fieldCode="EL" term="%22Primary+Education%22">Primary Education</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+1%22">Grade 1</searchLink><br /><searchLink fieldCode="EL" term="%22Grade+2%22">Grade 2</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Electronic+Learning%22">Electronic Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Tutoring%22">Tutoring</searchLink><br /><searchLink fieldCode="DE" term="%22Student+Characteristics%22">Student Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Institutional+Characteristics%22">Institutional Characteristics</searchLink><br /><searchLink fieldCode="DE" term="%22Public+Schools%22">Public Schools</searchLink><br /><searchLink fieldCode="DE" term="%22Kindergarten%22">Kindergarten</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+1%22">Grade 1</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+2%22">Grade 2</searchLink><br /><searchLink fieldCode="DE" term="%22Program+Effectiveness%22">Program Effectiveness</searchLink><br /><searchLink fieldCode="DE" term="%22Emergent+Literacy%22">Emergent Literacy</searchLink><br /><searchLink fieldCode="DE" term="%22Literacy+Education%22">Literacy Education</searchLink><br /><searchLink fieldCode="DE" term="%22Time+Factors+%28Learning%29%22">Time Factors (Learning)</searchLink><br /><searchLink fieldCode="DE" term="%22Classification%22">Classification</searchLink>
– Name: Subject
  Label: Geographic Terms
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Texas%22">Texas</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background/Context: Virtual tutoring has grown significantly as a personalized learning tool, especially in response to the academic disruptions caused by the COVID-19 pandemic. While out-of-school virtual tutoring has demonstrated moderate to large positive effects on student learning (0.20-0.40 SDs; Carlana and La Ferrara, 2024; Deacon and Chojnacki, 2023; Gortazar et al., 2024; Roschelle et al., 2020), in-school virtual tutoring shows significantly smaller impacts (0.01-0.08 SDs; Kraft et al., 2022; Ready et al., 2024; Robinson et al., 2024). Although many factors may explain this discrepancy, we hypothesize that in-school virtual tutoring can be more susceptible to disruptions similar to classroom instruction (Kraft and Monti-Nussbaum, 2021), affecting the overall instruction time. Prior research has shown that instructional time is a critical determinant of learning outcomes (Yesil Dagli (2019); Gromada and Shewbridge (2016); Holland et al. (2015); Kraft and Novicoff (2024); Patall et al. (2010), but due to difficulties in data collection, there has been limited empirical work quantifying how disruptions can reduce instruction time and affect learning. Purpose/Objective/Research Question: The primary objective of this study is to investigate how much instructional time is lost due to disruptions during in-school virtual tutoring sessions and to analyze the relationship between these disruptions and student and school characteristics. Specifically, the study aims to identify different types of disruptions and quantify the time they consume during tutoring, using a text classification model trained on tutor transcripts annotated by humans. The underlying hypothesis is that a substantial portion of virtual tutoring time in schools is spent addressing disruptions, which contributes to the diminished effectiveness observed in such settings. Setting: Our study uses data collected for the RCT conducted in Robinson et al., 2024. The virtual tutoring intervention occurred across 12 campuses within the same public school system in Texas. Tutoring sessions were administered virtually during the school day, using a platform that connected students with remote tutors nationwide to deliver early literacy instruction to participating students assigned to one of the treatment groups. Population/Participants/Subjects: Data was collected from 1,357 students in kindergarten through second grade and 192 tutors who participated in an RCT to evaluate the effectiveness of in-school virtual tutoring (Robinson et al., 2024). Over 26,000 tutoring sessions were analyzed, with 19,448 individual tutoring sessions (1:1 student-tutor ratio) and 7,150 paired tutoring sessions (2:1 tutor-student ratio). Students were diverse in terms of demographics and learning needs, including those receiving free or reduced lunch, English learners, and students with disabilities. Tutoring was focused on early literacy development, and the main outcomes for the RCT were achievement scores on DIBELS and MAP Reading. Intervention/Program/Practice: The tutoring intervention involved early literacy virtual instruction delivered via a video conferencing platform. Tutors and students engaged in individualized or paired tutoring sessions using structured literacy materials. Sessions were recorded, and transcript and audio data were collected. A typical session lasted around 20-30 minutes, and the program implementation occurred from November 2022 to May 2023. Research Design: We use natural language processing (NLP) to classify and analyze types of disruptions during virtual tutoring. We developed a custom multi-class text classifier trained on tutor utterances labeled by human annotators. Sessions were selected from a two-week window in April 2023, and regression models were used to assess the relationship between session characteristics related to the students and their school and the prevalence of different types of disruptions. Data Collection and Analysis: We transcribed all 26,714 session recordings from the RCT's tutoring sessions for this study. We created a human-annotated set of 2,501 annotations of 1,488 unique utterances from 255 sessions to train our classifier. We achieved an inter-rater reliability of 0.71 using Krippendorff's alpha across 19 human annotators. Disruption categories included tech problems, student disruptions, background noise, and issues involving additional students or substitute tutors. A RoBERTa-based classifier was fine-tuned using 80% of the labeled data, with oversampling of less frequent categories to address class imbalance. Accuracy reached 74% overall (80% for disruption vs. non-disruption). We estimated disruption durations based on the classifier labeling of each utterance and utterance timestamps. Our analysis used session-level aggregation of disruption time, examining how disruption time varied by student composition (e.g., 1:1 vs. 2:1 tutoring), demographics, and school for the students in the session. We estimate the following regression, where i is the session, y_i is the percentage of session time spent on one of the categories from our disruption classifier, 2:1 is a dummy for students assigned to paired tutoring (instead of individual), X represents a vector of student demographics and other characteristics, [delta] represents school (s) fixed effects, and represents grade fixed effects. y[subscript i] = [beta subscript 0] + [beta subscript 1] (2 : 1t)[subscript i] + X[subscript i][gamma] + [delta subscript s(i)] + [lambda subscript g(i)] + [epsilon subscript i]. Findings/Results: We found that approximately 25% of tutoring time was spent managing disruptions. Of this, 11.8% of time was attributed to student behavior, and 9% to technology issues. Pair tutoring (2:1) sessions were associated with significantly less usable instructional time--nearly 9 percentage points lower than individual (1:1) sessions. Sessions with at least one female student had more usable instructional time (+3 p.p.), suggesting possible behavioral differences by gender. Tech issues were less prevalent among second-grade students, likely due to their greater experience with technology. We also found that tech issues and background noise disruptions correlated more strongly with specific schools, suggesting variations in the implementation environment affected the amount of instruction time available to students through virtual tutoring. Conclusions: Disruptions are a significant drain on instructional time in in-school virtual tutoring settings, with both student behavior and environmental factors contributing. Notably, disruptions varied systematically by session type, student demographics, and school, suggesting that targeted program design and implementation changes--such as quieter tutoring locations, better tech support, and smaller group sizes--could mitigate some of these effects. As we continue to work on this study, our next steps include extending our analysis to the full set of tutoring sessions and investigating whether lower levels of school environment disruptions, such as tech issues and background noise, also correlate with larger learning gains for students by estimating differential impact on student performance on the end-of-year DIBELS assessment.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: URL
  Label: Access URL
  Group: URL
  Data: <link linkTarget="URL" linkTerm="https://www.sree.org/2025-conference" linkWindow="_blank">https://www.sree.org/2025-conference</link>
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED677776
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED677776
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    Subjects:
      – SubjectFull: Electronic Learning
        Type: general
      – SubjectFull: Tutoring
        Type: general
      – SubjectFull: Student Characteristics
        Type: general
      – SubjectFull: Institutional Characteristics
        Type: general
      – SubjectFull: Public Schools
        Type: general
      – SubjectFull: Kindergarten
        Type: general
      – SubjectFull: Grade 1
        Type: general
      – SubjectFull: Grade 2
        Type: general
      – SubjectFull: Program Effectiveness
        Type: general
      – SubjectFull: Emergent Literacy
        Type: general
      – SubjectFull: Literacy Education
        Type: general
      – SubjectFull: Time Factors (Learning)
        Type: general
      – SubjectFull: Classification
        Type: general
      – SubjectFull: Texas
        Type: general
    Titles:
      – TitleFull: Understanding Disruptions to Virtual Learning: Causes of and Variation in Lost Instructional Time
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Society for Research on Educational Effectiveness (SREE)
      – PersonEntity:
          Name:
            NameFull: Xander Beberman
      – PersonEntity:
          Name:
            NameFull: Sarah Novicoff
      – PersonEntity:
          Name:
            NameFull: Ana Trindade Ribeiro
      – PersonEntity:
          Name:
            NameFull: Carly Robinson
      – PersonEntity:
          Name:
            NameFull: Susanna Loeb
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 11
              M: 10
              Type: published
              Y: 2025
          Titles:
            – TitleFull: Society for Research on Educational Effectiveness
              Type: main
ResultId 1