Superficially Plausible Outputs from a Black Box: Problematising GenAI Tools for Analysing Qualitative SoTL Data
Saved in:
| Title: | Superficially Plausible Outputs from a Black Box: Problematising GenAI Tools for Analysing Qualitative SoTL Data |
|---|---|
| Language: | English |
| Authors: | Mirjam Sophia Glessmer, Rachel Forsyth |
| Source: | Teaching & Learning Inquiry. 2025 13. |
| Availability: | University of Calgary. Libraries & Cultural Resources, 410 University Court NW, Calgary, Alberta, T2N 1N4, Canada. Tel: 403-220-7175; e-mail: TLI@ucalgary.ca; Web site: https://journalhosting.ucalgary.ca/index.php/TLI/index |
| Peer Reviewed: | Y |
| Page Count: | 9 |
| Publication Date: | 2025 |
| Document Type: | Journal Articles Reports - Research |
| Descriptors: | Artificial Intelligence, Research Methodology, Data Analysis, Scholarship, Instruction, Learning, Reliability, Validity, Ethics, Educational Development, Algorithms |
| ISSN: | 2167-4779 2167-4787 |
| Abstract: | Generative AI tools (GenAI) are increasingly used for academic tasks, including qualitative data analysis for the Scholarship of Teaching and Learning (SoTL). In our practice as academic developers, we are frequently asked for advice on whether this use for GenAI is reliable, valid, and ethical. Since this is a new field, we have not been able to answer this confidently based on published literature, which depicts both very positive as well as highly cautionary accounts. To fill this gap, we experiment with the use of chatbot style GenAI (namely ChatGPT 4, ChatGPT 4o, and Microsoft Copilot) to support or conduct qualitative analysis of survey and interview data from a SoTL project, which had previously been analysed by experienced researchers using thematic analysis. At first sight, the output looked plausible, but the results were incomplete and not reproducible. In some instances, interpretations and extrapolations of data happened when it was clearly stated in the prompt that the tool should only analyse a specified dataset based on explicit instructions. Since both algorithm and training data of the GenAI tools are undisclosed, it is impossible to know how the outputs had been arrived at. We conclude that while results may look plausible initially, digging deeper soon reveals serious problems; the lack of transparency about how analyses are conducted and results are generated means that no reproducible method can be described. We therefore warn against an uncritical use of GenAI in qualitative analysis of SoTL data. |
| Abstractor: | As Provided |
| Entry Date: | 2025 |
| Accession Number: | EJ1460022 |
| Database: | ERIC |
| FullText | Text: Availability: 0 CustomLinks: – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1460022 Name: ERIC Full Text Category: fullText Text: Full Text from ERIC |
|---|---|
| Header | DbId: eric DbLabel: ERIC An: EJ1460022 AccessLevel: 3 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Superficially Plausible Outputs from a Black Box: Problematising GenAI Tools for Analysing Qualitative SoTL Data – Name: Language Label: Language Group: Lang Data: English – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Mirjam+Sophia+Glessmer%22">Mirjam Sophia Glessmer</searchLink><br /><searchLink fieldCode="AR" term="%22Rachel+Forsyth%22">Rachel Forsyth</searchLink> – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="SO" term="%22Teaching+%26+Learning+Inquiry%22"><i>Teaching & Learning Inquiry</i></searchLink>. 2025 13. – Name: Avail Label: Availability Group: Avail Data: University of Calgary. Libraries & Cultural Resources, 410 University Court NW, Calgary, Alberta, T2N 1N4, Canada. Tel: 403-220-7175; e-mail: TLI@ucalgary.ca; Web site: https://journalhosting.ucalgary.ca/index.php/TLI/index – Name: PeerReviewed Label: Peer Reviewed Group: SrcInfo Data: Y – Name: Pages Label: Page Count Group: Src Data: 9 – Name: DatePubCY Label: Publication Date Group: Date Data: 2025 – Name: TypeDocument Label: Document Type Group: TypDoc Data: Journal Articles<br />Reports - Research – Name: Subject Label: Descriptors Group: Su Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Research+Methodology%22">Research Methodology</searchLink><br /><searchLink fieldCode="DE" term="%22Data+Analysis%22">Data Analysis</searchLink><br /><searchLink fieldCode="DE" term="%22Scholarship%22">Scholarship</searchLink><br /><searchLink fieldCode="DE" term="%22Instruction%22">Instruction</searchLink><br /><searchLink fieldCode="DE" term="%22Learning%22">Learning</searchLink><br /><searchLink fieldCode="DE" term="%22Reliability%22">Reliability</searchLink><br /><searchLink fieldCode="DE" term="%22Validity%22">Validity</searchLink><br /><searchLink fieldCode="DE" term="%22Ethics%22">Ethics</searchLink><br /><searchLink fieldCode="DE" term="%22Educational+Development%22">Educational Development</searchLink><br /><searchLink fieldCode="DE" term="%22Algorithms%22">Algorithms</searchLink> – Name: ISSN Label: ISSN Group: ISSN Data: 2167-4779<br />2167-4787 – Name: Abstract Label: Abstract Group: Ab Data: Generative AI tools (GenAI) are increasingly used for academic tasks, including qualitative data analysis for the Scholarship of Teaching and Learning (SoTL). In our practice as academic developers, we are frequently asked for advice on whether this use for GenAI is reliable, valid, and ethical. Since this is a new field, we have not been able to answer this confidently based on published literature, which depicts both very positive as well as highly cautionary accounts. To fill this gap, we experiment with the use of chatbot style GenAI (namely ChatGPT 4, ChatGPT 4o, and Microsoft Copilot) to support or conduct qualitative analysis of survey and interview data from a SoTL project, which had previously been analysed by experienced researchers using thematic analysis. At first sight, the output looked plausible, but the results were incomplete and not reproducible. In some instances, interpretations and extrapolations of data happened when it was clearly stated in the prompt that the tool should only analyse a specified dataset based on explicit instructions. Since both algorithm and training data of the GenAI tools are undisclosed, it is impossible to know how the outputs had been arrived at. We conclude that while results may look plausible initially, digging deeper soon reveals serious problems; the lack of transparency about how analyses are conducted and results are generated means that no reproducible method can be described. We therefore warn against an uncritical use of GenAI in qualitative analysis of SoTL data. – Name: AbstractInfo Label: Abstractor Group: Ab Data: As Provided – Name: DateEntry Label: Entry Date Group: Date Data: 2025 – Name: AN Label: Accession Number Group: ID Data: EJ1460022 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1460022 |
| RecordInfo | BibRecord: BibEntity: Languages: – Text: English PhysicalDescription: Pagination: PageCount: 9 Subjects: – SubjectFull: Artificial Intelligence Type: general – SubjectFull: Research Methodology Type: general – SubjectFull: Data Analysis Type: general – SubjectFull: Scholarship Type: general – SubjectFull: Instruction Type: general – SubjectFull: Learning Type: general – SubjectFull: Reliability Type: general – SubjectFull: Validity Type: general – SubjectFull: Ethics Type: general – SubjectFull: Educational Development Type: general – SubjectFull: Algorithms Type: general Titles: – TitleFull: Superficially Plausible Outputs from a Black Box: Problematising GenAI Tools for Analysing Qualitative SoTL Data Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Mirjam Sophia Glessmer – PersonEntity: Name: NameFull: Rachel Forsyth IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2025 Identifiers: – Type: issn-print Value: 2167-4779 – Type: issn-electronic Value: 2167-4787 Numbering: – Type: volume Value: 13 Titles: – TitleFull: Teaching & Learning Inquiry Type: main |
| ResultId | 1 |
