View in EDS

Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool

Saved in:
Bibliographic Details
Title: Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool
Language: English
Authors: Andrew Potter, Zeinab Serhan, Nishad A. Patne, Püren Öncel, Ishrat Ahm, Tracy Arner, Rezwana Islam, Rod D. Roscoe, Laura A. Allen, Scott A. Crossley, Danielle S. McNamara
Source: Journal of Educational Data Mining. 2026 18(1):113-155.
Availability: International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM
Peer Reviewed: Y
Page Count: 43
Publication Date: 2026
Sponsoring Agency: Institute of Education Sciences (ED)
Contract Number: R305A180261
Document Type: Journal Articles
Reports - Research
Education Level: Higher Education
Postsecondary Education
Descriptors: Higher Education, Introductory Courses, Writing (Composition), Writing Instruction, Writing Evaluation, Artificial Intelligence, Natural Language Processing, Internet, Computer Assisted Instruction, Computer Uses in Education, Feedback (Response)
ISSN: 2157-2100
Abstract: This study introduces a hybrid human-AI workflow to qualitative data analysis within the participatory design of the Writing Analytics Toolkit (WAT), an open-source platform that provides formative feedback on student writing using natural language processing. The toolkit includes a classroom-facing implementation (WAT Classroom; WAT-C), designed to support instruction, and a researcher-facing implementation (WAT Researcher; WAT-R), designed to support analytic and validation workflows. Nine experienced college writing instructors (with 97 cumulative years of teaching) participated in focus group sessions to evaluate an early prototype of the classroom version of WAT (WAT-C), offering formative input on usability, instructional alignment, and feedback clarity. To analyze the resulting qualitative data, we employed a novel AI-augmented analytic process: GPT-4o, integrated within a secure, retrieval-augmented system, to generate inductive codes and preliminary themes from transcripts. These AI-generated outputs were iteratively reviewed, critiqued, refined, and synthesized by researchers, supporting both analytical scalability and interpretive rigor. This human-AI partnership enabled efficient thematic exploration while preserving methodological transparency and researcher judgment. Findings from both qualitative and complementary survey data identified four key design priorities: (1) clearer, more concise feedback, (2) increased instructor customization, (3) reduced administrative burden, and (4) a simplified user interface. These insights directly informed subsequent revisions to WAT-C, including a redesigned feedback interface, customizable metric targets, learning management system integration, and a more intuitive layout. This work illustrates how large language models (LLMs) can support inductive qualitative analysis within participatory design workflows. Moreover, results demonstrate how this workflow can inform iterative educational technology development. Implications include the need to ensure ethical oversight, researcher-led interpretation, and alignment with instructional priorities when incorporating AI into the design of educational technologies.
Abstractor: As Provided
IES Funded: Yes
Entry Date: 2026
Accession Number: EJ1506627
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1506627
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1506627
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Andrew+Potter%22">Andrew Potter</searchLink><br /><searchLink fieldCode="AR" term="%22Zeinab+Serhan%22">Zeinab Serhan</searchLink><br /><searchLink fieldCode="AR" term="%22Nishad+A%2E+Patne%22">Nishad A. Patne</searchLink><br /><searchLink fieldCode="AR" term="%22Püren+Öncel%22">Püren Öncel</searchLink><br /><searchLink fieldCode="AR" term="%22Ishrat+Ahm%22">Ishrat Ahm</searchLink><br /><searchLink fieldCode="AR" term="%22Tracy+Arner%22">Tracy Arner</searchLink><br /><searchLink fieldCode="AR" term="%22Rezwana+Islam%22">Rezwana Islam</searchLink><br /><searchLink fieldCode="AR" term="%22Rod+D%2E+Roscoe%22">Rod D. Roscoe</searchLink><br /><searchLink fieldCode="AR" term="%22Laura+A%2E+Allen%22">Laura A. Allen</searchLink><br /><searchLink fieldCode="AR" term="%22Scott+A%2E+Crossley%22">Scott A. Crossley</searchLink><br /><searchLink fieldCode="AR" term="%22Danielle+S%2E+McNamara%22">Danielle S. McNamara</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Data+Mining%22"><i>Journal of Educational Data Mining</i></searchLink>. 2026 18(1):113-155.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 43
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: SourceSuprt
  Label: Sponsoring Agency
  Group: SrcSuprt
  Data: Institute of Education Sciences (ED)
– Name: NumberContract
  Label: Contract Number
  Group: NumCntrct
  Data: R305A180261
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="EL" term="%22Postsecondary+Education%22">Postsecondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Higher+Education%22">Higher Education</searchLink><br /><searchLink fieldCode="DE" term="%22Introductory+Courses%22">Introductory Courses</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+%28Composition%29%22">Writing (Composition)</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Instruction%22">Writing Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Writing+Evaluation%22">Writing Evaluation</searchLink><br /><searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Internet%22">Internet</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Assisted+Instruction%22">Computer Assisted Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Computer+Uses+in+Education%22">Computer Uses in Education</searchLink><br /><searchLink fieldCode="DE" term="%22Feedback+%28Response%29%22">Feedback (Response)</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2157-2100
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: This study introduces a hybrid human-AI workflow to qualitative data analysis within the participatory design of the Writing Analytics Toolkit (WAT), an open-source platform that provides formative feedback on student writing using natural language processing. The toolkit includes a classroom-facing implementation (WAT Classroom; WAT-C), designed to support instruction, and a researcher-facing implementation (WAT Researcher; WAT-R), designed to support analytic and validation workflows. Nine experienced college writing instructors (with 97 cumulative years of teaching) participated in focus group sessions to evaluate an early prototype of the classroom version of WAT (WAT-C), offering formative input on usability, instructional alignment, and feedback clarity. To analyze the resulting qualitative data, we employed a novel AI-augmented analytic process: GPT-4o, integrated within a secure, retrieval-augmented system, to generate inductive codes and preliminary themes from transcripts. These AI-generated outputs were iteratively reviewed, critiqued, refined, and synthesized by researchers, supporting both analytical scalability and interpretive rigor. This human-AI partnership enabled efficient thematic exploration while preserving methodological transparency and researcher judgment. Findings from both qualitative and complementary survey data identified four key design priorities: (1) clearer, more concise feedback, (2) increased instructor customization, (3) reduced administrative burden, and (4) a simplified user interface. These insights directly informed subsequent revisions to WAT-C, including a redesigned feedback interface, customizable metric targets, learning management system integration, and a more intuitive layout. This work illustrates how large language models (LLMs) can support inductive qualitative analysis within participatory design workflows. Moreover, results demonstrate how this workflow can inform iterative educational technology development. Implications include the need to ensure ethical oversight, researcher-led interpretation, and alignment with instructional priorities when incorporating AI into the design of educational technologies.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: CodeSource
  Label: IES Funded
  Group: SrcInfo
  Data: Yes
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1506627
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1506627
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 43
        StartPage: 113
    Subjects:
      – SubjectFull: Higher Education
        Type: general
      – SubjectFull: Introductory Courses
        Type: general
      – SubjectFull: Writing (Composition)
        Type: general
      – SubjectFull: Writing Instruction
        Type: general
      – SubjectFull: Writing Evaluation
        Type: general
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Internet
        Type: general
      – SubjectFull: Computer Assisted Instruction
        Type: general
      – SubjectFull: Computer Uses in Education
        Type: general
      – SubjectFull: Feedback (Response)
        Type: general
    Titles:
      – TitleFull: Human-AI Collaboration for Qualitative Analysis in Participatory Design: Refining the Writing Analytics Tool
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Andrew Potter
      – PersonEntity:
          Name:
            NameFull: Zeinab Serhan
      – PersonEntity:
          Name:
            NameFull: Nishad A. Patne
      – PersonEntity:
          Name:
            NameFull: Püren Öncel
      – PersonEntity:
          Name:
            NameFull: Ishrat Ahm
      – PersonEntity:
          Name:
            NameFull: Tracy Arner
      – PersonEntity:
          Name:
            NameFull: Rezwana Islam
      – PersonEntity:
          Name:
            NameFull: Rod D. Roscoe
      – PersonEntity:
          Name:
            NameFull: Laura A. Allen
      – PersonEntity:
          Name:
            NameFull: Scott A. Crossley
      – PersonEntity:
          Name:
            NameFull: Danielle S. McNamara
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 2157-2100
          Numbering:
            – Type: volume
              Value: 18
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Educational Data Mining
              Type: main
ResultId 1