View in EDS

I Don't Have Time! But Keep Me in the Loop: Co-Designing Requirements for a Learning Analytics Cockpit with Teachers

Saved in:
Bibliographic Details
Title: I Don't Have Time! But Keep Me in the Loop: Co-Designing Requirements for a Learning Analytics Cockpit with Teachers
Language: English
Authors: Onur Karademir (ORCID 0000-0002-6985-4202), Daniele Di Mitri (ORCID 0000-0002-9331-6893), Jan Schneider (ORCID 0000-0001-8578-6409), Ioana Jivet (ORCID 0000-0002-8715-2642), Jörn Allmang, Sebastian Gombert (ORCID 0000-0001-5598-9547), Marcus Kubsch (ORCID 0000-0001-5497-8336), Knut Neumann (ORCID 0000-0002-4391-7308), Hendrik Drachsler (ORCID 0000-0001-8407-5314)
Source: Journal of Computer Assisted Learning. 2024 40(6):2681-2699.
Availability: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Peer Reviewed: Y
Page Count: 19
Publication Date: 2024
Document Type: Journal Articles
Reports - Research
Education Level: Secondary Education
Descriptors: Learning Analytics, Secondary School Teachers, Educational Technology, Design, Visualization, Formative Evaluation, Feedback (Response), STEM Education, Automation, Faculty Workload
DOI: 10.1111/jcal.12997
ISSN: 0266-4909
1365-2729
Abstract: Background: Teacher dashboards can help secondary school teachers manage online learning activities and inform instructional decisions by visualising information about class learning. However, when designing teacher dashboards, it is not trivial to choose which information to display, because not all of the vast amount of information retrieved from digital learning environments is useful for teaching. Information elicited from formative assessment (FA), though, is a strong predictor for student performance and can be a useful data source for effective teacher dashboards. Especially in the secondary education context, FA and feedback on FA, have been extensively studied and shown to positively affect student learning outcomes. Moreover, secondary teachers struggle to make sense of the information displayed in dashboards and decide on pedagogical actions, such as providing feedback to students. Objectives: To facilitate the provision of feedback for secondary school teachers via a teacher dashboard, this study identifies requirements for designing a Learning Analytics Cockpit (LA Cockpit), that is, (1) a teacher dashboard that provides teachers with visualisations of results from formative assessment (FA) and (2) a feedback system that supports teachers in providing feedback to students. Methods: This study was conducted in the context of STEM classes and is based on semi-structured co-design interviews with German secondary school teachers. In these interviews, we first explored challenges teachers encountered in monitoring students' learning and providing feedback. Second, in the ideation phase, teachers were asked to define features an LA Cockpit for FA should have. Finally, in the evaluation phase, we provided teachers with a design template for an LA Cockpit, the LAC_Template, which was built upon our previous work and feedback theory, and asked them to evaluate and improve it. Further design requirements were derived based on the evaluation of the LAC_Template and teachers' suggestions for improvement. Results: We derived 16 requirements for designing an LA Cockpit for FA in secondary schools. Findings from the interviews indicated that the feedback system of an LA Cockpit should address teachers' time limitations in giving students individualised feedback. It should therefore be designed to minimise the steps required to deliver feedback. To reduce workload, teachers requested an automated reminder to send feedback, but with the ability to adjust feedback to the learning context. Such a semi-automated feedback system can help teachers support students individually but also underline the importance of actively involving teachers in the feedback loop and giving them control when using such technologies in secondary school practice. A challenge for future teacher dashboard designs could be to find a balance between technology and teacher control that utilises the strengths of both in a beneficial combination.
Abstractor: As Provided
Entry Date: 2024
Accession Number: EJ1448435
Database: ERIC
FullText Text:
  Availability: 0
Header DbId: eric
DbLabel: ERIC
An: EJ1448435
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: I Don't Have Time! But Keep Me in the Loop: Co-Designing Requirements for a Learning Analytics Cockpit with Teachers
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Onur+Karademir%22">Onur Karademir</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-6985-4202">0000-0002-6985-4202</externalLink>)<br /><searchLink fieldCode="AR" term="%22Daniele+Di+Mitri%22">Daniele Di Mitri</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-9331-6893">0000-0002-9331-6893</externalLink>)<br /><searchLink fieldCode="AR" term="%22Jan+Schneider%22">Jan Schneider</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-8578-6409">0000-0001-8578-6409</externalLink>)<br /><searchLink fieldCode="AR" term="%22Ioana+Jivet%22">Ioana Jivet</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-8715-2642">0000-0002-8715-2642</externalLink>)<br /><searchLink fieldCode="AR" term="%22Jörn+Allmang%22">Jörn Allmang</searchLink><br /><searchLink fieldCode="AR" term="%22Sebastian+Gombert%22">Sebastian Gombert</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-5598-9547">0000-0001-5598-9547</externalLink>)<br /><searchLink fieldCode="AR" term="%22Marcus+Kubsch%22">Marcus Kubsch</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-5497-8336">0000-0001-5497-8336</externalLink>)<br /><searchLink fieldCode="AR" term="%22Knut+Neumann%22">Knut Neumann</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0002-4391-7308">0000-0002-4391-7308</externalLink>)<br /><searchLink fieldCode="AR" term="%22Hendrik+Drachsler%22">Hendrik Drachsler</searchLink> (ORCID <externalLink term="https://orcid.org/0000-0001-8407-5314">0000-0001-8407-5314</externalLink>)
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Computer+Assisted+Learning%22"><i>Journal of Computer Assisted Learning</i></searchLink>. 2024 40(6):2681-2699.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 19
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2024
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Learning+Analytics%22">Learning Analytics</searchLink><br /><searchLink fieldCode="DE" term="%22Secondary+School+Teachers%22">Secondary School Teachers</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Technology%22">Educational Technology</searchLink><br /><searchLink fieldCode="DE" term="%22Design%22">Design</searchLink><br /><searchLink fieldCode="DE" term="%22Visualization%22">Visualization</searchLink><br /><searchLink fieldCode="DE" term="%22Formative+Evaluation%22">Formative Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink><br /><searchLink fieldCode="DE" term="%22STEM+Education%22">STEM Education</searchLink><br /><searchLink fieldCode="DE" term="%22Automation%22">Automation</searchLink><br /><searchLink fieldCode="DE" term="%22Faculty+Workload%22">Faculty Workload</searchLink>
– Name: DOI
  Label: DOI
  Group: ID
  Data: 10.1111/jcal.12997
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 0266-4909<br />1365-2729
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Background: Teacher dashboards can help secondary school teachers manage online learning activities and inform instructional decisions by visualising information about class learning. However, when designing teacher dashboards, it is not trivial to choose which information to display, because not all of the vast amount of information retrieved from digital learning environments is useful for teaching. Information elicited from formative assessment (FA), though, is a strong predictor for student performance and can be a useful data source for effective teacher dashboards. Especially in the secondary education context, FA and feedback on FA, have been extensively studied and shown to positively affect student learning outcomes. Moreover, secondary teachers struggle to make sense of the information displayed in dashboards and decide on pedagogical actions, such as providing feedback to students. Objectives: To facilitate the provision of feedback for secondary school teachers via a teacher dashboard, this study identifies requirements for designing a Learning Analytics Cockpit (LA Cockpit), that is, (1) a teacher dashboard that provides teachers with visualisations of results from formative assessment (FA) and (2) a feedback system that supports teachers in providing feedback to students. Methods: This study was conducted in the context of STEM classes and is based on semi-structured co-design interviews with German secondary school teachers. In these interviews, we first explored challenges teachers encountered in monitoring students' learning and providing feedback. Second, in the ideation phase, teachers were asked to define features an LA Cockpit for FA should have. Finally, in the evaluation phase, we provided teachers with a design template for an LA Cockpit, the LAC_Template, which was built upon our previous work and feedback theory, and asked them to evaluate and improve it. Further design requirements were derived based on the evaluation of the LAC_Template and teachers' suggestions for improvement. Results: We derived 16 requirements for designing an LA Cockpit for FA in secondary schools. Findings from the interviews indicated that the feedback system of an LA Cockpit should address teachers' time limitations in giving students individualised feedback. It should therefore be designed to minimise the steps required to deliver feedback. To reduce workload, teachers requested an automated reminder to send feedback, but with the ability to adjust feedback to the learning context. Such a semi-automated feedback system can help teachers support students individually but also underline the importance of actively involving teachers in the feedback loop and giving them control when using such technologies in secondary school practice. A challenge for future teacher dashboard designs could be to find a balance between technology and teacher control that utilises the strengths of both in a beneficial combination.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2024
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1448435
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1448435
RecordInfo BibRecord:
  BibEntity:
    Identifiers:
      – Type: doi
        Value: 10.1111/jcal.12997
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 19
        StartPage: 2681
    Subjects:
      – SubjectFull: Learning Analytics
        Type: general
      – SubjectFull: Secondary School Teachers
        Type: general
      – SubjectFull: Educational Technology
        Type: general
      – SubjectFull: Design
        Type: general
      – SubjectFull: Visualization
        Type: general
      – SubjectFull: Formative Evaluation
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
      – SubjectFull: STEM Education
        Type: general
      – SubjectFull: Automation
        Type: general
      – SubjectFull: Faculty Workload
        Type: general
    Titles:
      – TitleFull: I Don't Have Time! But Keep Me in the Loop: Co-Designing Requirements for a Learning Analytics Cockpit with Teachers
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Onur Karademir
      – PersonEntity:
          Name:
            NameFull: Daniele Di Mitri
      – PersonEntity:
          Name:
            NameFull: Jan Schneider
      – PersonEntity:
          Name:
            NameFull: Ioana Jivet
      – PersonEntity:
          Name:
            NameFull: Jörn Allmang
      – PersonEntity:
          Name:
            NameFull: Sebastian Gombert
      – PersonEntity:
          Name:
            NameFull: Marcus Kubsch
      – PersonEntity:
          Name:
            NameFull: Knut Neumann
      – PersonEntity:
          Name:
            NameFull: Hendrik Drachsler
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 12
              Type: published
              Y: 2024
          Identifiers:
            – Type: issn-print
              Value: 0266-4909
            – Type: issn-electronic
              Value: 1365-2729
          Numbering:
            – Type: volume
              Value: 40
            – Type: issue
              Value: 6
          Titles:
            – TitleFull: Journal of Computer Assisted Learning
              Type: main
ResultId 1