View in EDS

Evaluating the Explainers: Black-Box Explainable Machine Learning for Student Success Prediction in MOOCs

Saved in:
Bibliographic Details
Title: Evaluating the Explainers: Black-Box Explainable Machine Learning for Student Success Prediction in MOOCs
Language: English
Authors: Swamy, Vinitra, Radmehr, Bahar, Krco, Natasa, Marras, Mirko, Käser, Tanja
Source: International Educational Data Mining Society. 2022.
Availability: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
Peer Reviewed: Y
Page Count: 12
Publication Date: 2022
Document Type: Speeches/Meeting Papers
Reports - Research
Descriptors: Artificial Intelligence, Academic Achievement, Grade Prediction, MOOCs, Teaching Methods, Models
Abstract: Neural networks are ubiquitous in applied machine learning for education. Their pervasive success in predictive performance comes alongside a severe weakness, the lack of explainability of their decisions, especially relevant in humancentric fields. We implement five state-of-the-art methodologies for explaining black-box machine learning models (LIME, PermutationSHAP, KernelSHAP, DiCE, CEM) and examine the strengths of each approach on the downstream task of student performance prediction for five massive open online courses. Our experiments demonstrate that the families of explainers do not agree with each other on feature importance for the same Bidirectional LSTM models with the same representative set of students. We use Principal Component Analysis, Jensen-Shannon distance, and Spearman's rank-order correlation to quantitatively cross-examine explanations across methods and courses. Furthermore, we validate explainer performance across curriculum-based prerequisite relationships. Our results come to the concerning conclusion that the choice of explainer is an important decision and is in fact paramount to the interpretation of the predictive results, even more so than the course the model is trained on. Source code and models are released at http://github.com/epfl-ml4ed/evaluating-explainers. [For the full proceedings, see ED623995.]
Abstractor: As Provided
Entry Date: 2022
Accession Number: ED624070
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED624070
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: ED624070
AccessLevel: 3
PubType: Conference
PubTypeId: conference
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Evaluating the Explainers: Black-Box Explainable Machine Learning for Student Success Prediction in MOOCs
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Swamy%2C+Vinitra%22">Swamy, Vinitra</searchLink><br /><searchLink fieldCode="AR" term="%22Radmehr%2C+Bahar%22">Radmehr, Bahar</searchLink><br /><searchLink fieldCode="AR" term="%22Krco%2C+Natasa%22">Krco, Natasa</searchLink><br /><searchLink fieldCode="AR" term="%22Marras%2C+Mirko%22">Marras, Mirko</searchLink><br /><searchLink fieldCode="AR" term="%22Käser%2C+Tanja%22">Käser, Tanja</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22International+Educational+Data+Mining+Society%22"><i>International Educational Data Mining Society</i></searchLink>. 2022.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Educational Data Mining Society. e-mail: admin@educationaldatamining.org; Web site: https://educationaldatamining.org/conferences/
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 12
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2022
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Speeches/Meeting Papers<br />Reports - Research
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Academic+Achievement%22">Academic Achievement</searchLink><br /><searchLink fieldCode="DE" term="%22Grade+Prediction%22">Grade Prediction</searchLink><br /><searchLink fieldCode="DE" term="%22MOOCs%22">MOOCs</searchLink><br /><searchLink fieldCode="DE" term="%22Teaching+Methods%22">Teaching Methods</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink>
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Neural networks are ubiquitous in applied machine learning for education. Their pervasive success in predictive performance comes alongside a severe weakness, the lack of explainability of their decisions, especially relevant in humancentric fields. We implement five state-of-the-art methodologies for explaining black-box machine learning models (LIME, PermutationSHAP, KernelSHAP, DiCE, CEM) and examine the strengths of each approach on the downstream task of student performance prediction for five massive open online courses. Our experiments demonstrate that the families of explainers do not agree with each other on feature importance for the same Bidirectional LSTM models with the same representative set of students. We use Principal Component Analysis, Jensen-Shannon distance, and Spearman's rank-order correlation to quantitatively cross-examine explanations across methods and courses. Furthermore, we validate explainer performance across curriculum-based prerequisite relationships. Our results come to the concerning conclusion that the choice of explainer is an important decision and is in fact paramount to the interpretation of the predictive results, even more so than the course the model is trained on. Source code and models are released at http://github.com/epfl-ml4ed/evaluating-explainers. [For the full proceedings, see ED623995.]
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2022
– Name: AN
  Label: Accession Number
  Group: ID
  Data: ED624070
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=ED624070
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 12
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Academic Achievement
        Type: general
      – SubjectFull: Grade Prediction
        Type: general
      – SubjectFull: MOOCs
        Type: general
      – SubjectFull: Teaching Methods
        Type: general
      – SubjectFull: Models
        Type: general
    Titles:
      – TitleFull: Evaluating the Explainers: Black-Box Explainable Machine Learning for Student Success Prediction in MOOCs
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Swamy, Vinitra
      – PersonEntity:
          Name:
            NameFull: Radmehr, Bahar
      – PersonEntity:
          Name:
            NameFull: Krco, Natasa
      – PersonEntity:
          Name:
            NameFull: Marras, Mirko
      – PersonEntity:
          Name:
            NameFull: Käser, Tanja
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2022
          Titles:
            – TitleFull: International Educational Data Mining Society
              Type: main
ResultId 1