View in EDS

Seeing Is Solving: MLLMs, Reasoning, and Refusal in Visual Math

Saved in:
Bibliographic Details
Title: Seeing Is Solving: MLLMs, Reasoning, and Refusal in Visual Math
Language: English
Authors: Ethan Croteau, Neil Heffernan
Source: Journal of Educational Data Mining. 2026 18(1):244-285.
Availability: International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM
Peer Reviewed: Y
Page Count: 42
Publication Date: 2026
Document Type: Journal Articles
Reports - Research
Education Level: Junior High Schools
Middle Schools
Secondary Education
Descriptors: Artificial Intelligence, Natural Language Processing, Models, Intelligent Tutoring Systems, Middle School Mathematics, Visual Aids, Graphs, Decision Making, Problem Solving, Computation, Accuracy, Error Patterns, Interrater Reliability
ISSN: 2157-2100
Abstract: Many middle--school math problems are image-dependent: the diagram or graph carries essential information. This matters for intelligent tutoring and accessibility, where systems must reason over figures and also decline responsibly when figures are missing. We evaluate six contemporary multimodal large language models (MLLMs)--three reasoning models and three non-reasoning models--on 376 Illustrative Mathematics (IM) "items" labeled as image-role "Required" (the figure contains task-critical information not recoverable from text alone without added assumptions). Each model attempts every item three times with and without the figure under a shared prompt and scoring protocol. To reduce image-role label subjectivity, we classify items as not "Required" when they are solvable from text alone without additional assumptions. With images, the top-performing reasoning models achieve accuracy in the mid-50%, while nonreasoning models fall in the mid-30s to low-40s. Without images, models overwhelmingly refuse rather than guess, with only rare correct-by-chance answers. Models show moderate agreement on which items are solvable, and we release two benchmark subsets of items solved consistently across models. A qualitative audit of 83 items shows that visual misreading is the dominant failure mode for non-reasoning models, while reasoning models more often produce correct answers accompanied by adequate explanations. These results suggest tutoring systems should gate automated scoring and learner-model updates on visual-evidence availability and use scaffolds that require explicit visual-evidence binding before algebra. For accessibility, systems should treat no-image refusals as missing-context signals and elicit the figure or a structured description, enabling description-substitution experiments. We release code, prompts, and summary artifacts for replication.
Abstractor: As Provided
Notes: https://osf.io/ct7bg
Entry Date: 2026
Accession Number: EJ1506390
Database: ERIC
FullText Text:
  Availability: 0
CustomLinks:
  – Url: https://eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=EJ1506390
    Name: ERIC Full Text
    Category: fullText
    Text: Full Text from ERIC
Header DbId: eric
DbLabel: ERIC
An: EJ1506390
AccessLevel: 3
PubType: Academic Journal
PubTypeId: academicJournal
PreciseRelevancyScore: 0
IllustrationInfo
Items – Name: Title
  Label: Title
  Group: Ti
  Data: Seeing Is Solving: MLLMs, Reasoning, and Refusal in Visual Math
– Name: Language
  Label: Language
  Group: Lang
  Data: English
– Name: Author
  Label: Authors
  Group: Au
  Data: <searchLink fieldCode="AR" term="%22Ethan+Croteau%22">Ethan Croteau</searchLink><br /><searchLink fieldCode="AR" term="%22Neil+Heffernan%22">Neil Heffernan</searchLink>
– Name: TitleSource
  Label: Source
  Group: Src
  Data: <searchLink fieldCode="SO" term="%22Journal+of+Educational+Data+Mining%22"><i>Journal of Educational Data Mining</i></searchLink>. 2026 18(1):244-285.
– Name: Avail
  Label: Availability
  Group: Avail
  Data: International Educational Data Mining. e-mail: jedm.editor@gmail.com; Web site: https://jedm.educationaldatamining.org/index.php/JEDM
– Name: PeerReviewed
  Label: Peer Reviewed
  Group: SrcInfo
  Data: Y
– Name: Pages
  Label: Page Count
  Group: Src
  Data: 42
– Name: DatePubCY
  Label: Publication Date
  Group: Date
  Data: 2026
– Name: TypeDocument
  Label: Document Type
  Group: TypDoc
  Data: Journal Articles<br />Reports - Research
– Name: Audience
  Label: Education Level
  Group: Audnce
  Data: <searchLink fieldCode="EL" term="%22Junior+High+Schools%22">Junior High Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Middle+Schools%22">Middle Schools</searchLink><br /><searchLink fieldCode="EL" term="%22Secondary+Education%22">Secondary Education</searchLink>
– Name: Subject
  Label: Descriptors
  Group: Su
  Data: <searchLink fieldCode="DE" term="%22Artificial+Intelligence%22">Artificial Intelligence</searchLink><br /><searchLink fieldCode="DE" term="%22Natural+Language+Processing%22">Natural Language Processing</searchLink><br /><searchLink fieldCode="DE" term="%22Models%22">Models</searchLink><br /><searchLink fieldCode="DE" term="%22Intelligent+Tutoring+Systems%22">Intelligent Tutoring Systems</searchLink><br /><searchLink fieldCode="DE" term="%22Middle+School+Mathematics%22">Middle School Mathematics</searchLink><br /><searchLink fieldCode="DE" term="%22Visual+Aids%22">Visual Aids</searchLink><br /><searchLink fieldCode="DE" term="%22Graphs%22">Graphs</searchLink><br /><searchLink fieldCode="DE" term="%22Decision+Making%22">Decision Making</searchLink><br /><searchLink fieldCode="DE" term="%22Problem+Solving%22">Problem Solving</searchLink><br /><searchLink fieldCode="DE" term="%22Computation%22">Computation</searchLink><br /><searchLink fieldCode="DE" term="%22Accuracy%22">Accuracy</searchLink><br /><searchLink fieldCode="DE" term="%22Error+Patterns%22">Error Patterns</searchLink><br /><searchLink fieldCode="DE" term="%22Interrater+Reliability%22">Interrater Reliability</searchLink>
– Name: ISSN
  Label: ISSN
  Group: ISSN
  Data: 2157-2100
– Name: Abstract
  Label: Abstract
  Group: Ab
  Data: Many middle--school math problems are image-dependent: the diagram or graph carries essential information. This matters for intelligent tutoring and accessibility, where systems must reason over figures and also decline responsibly when figures are missing. We evaluate six contemporary multimodal large language models (MLLMs)--three reasoning models and three non-reasoning models--on 376 Illustrative Mathematics (IM) "items" labeled as image-role "Required" (the figure contains task-critical information not recoverable from text alone without added assumptions). Each model attempts every item three times with and without the figure under a shared prompt and scoring protocol. To reduce image-role label subjectivity, we classify items as not "Required" when they are solvable from text alone without additional assumptions. With images, the top-performing reasoning models achieve accuracy in the mid-50%, while nonreasoning models fall in the mid-30s to low-40s. Without images, models overwhelmingly refuse rather than guess, with only rare correct-by-chance answers. Models show moderate agreement on which items are solvable, and we release two benchmark subsets of items solved consistently across models. A qualitative audit of 83 items shows that visual misreading is the dominant failure mode for non-reasoning models, while reasoning models more often produce correct answers accompanied by adequate explanations. These results suggest tutoring systems should gate automated scoring and learner-model updates on visual-evidence availability and use scaffolds that require explicit visual-evidence binding before algebra. For accessibility, systems should treat no-image refusals as missing-context signals and elicit the figure or a structured description, enabling description-substitution experiments. We release code, prompts, and summary artifacts for replication.
– Name: AbstractInfo
  Label: Abstractor
  Group: Ab
  Data: As Provided
– Name: Note
  Label: Notes
  Group: Note
  Data: https://osf.io/ct7bg
– Name: DateEntry
  Label: Entry Date
  Group: Date
  Data: 2026
– Name: AN
  Label: Accession Number
  Group: ID
  Data: EJ1506390
PLink https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=eric&AN=EJ1506390
RecordInfo BibRecord:
  BibEntity:
    Languages:
      – Text: English
    PhysicalDescription:
      Pagination:
        PageCount: 42
        StartPage: 244
    Subjects:
      – SubjectFull: Artificial Intelligence
        Type: general
      – SubjectFull: Natural Language Processing
        Type: general
      – SubjectFull: Models
        Type: general
      – SubjectFull: Intelligent Tutoring Systems
        Type: general
      – SubjectFull: Middle School Mathematics
        Type: general
      – SubjectFull: Visual Aids
        Type: general
      – SubjectFull: Graphs
        Type: general
      – SubjectFull: Decision Making
        Type: general
      – SubjectFull: Problem Solving
        Type: general
      – SubjectFull: Computation
        Type: general
      – SubjectFull: Accuracy
        Type: general
      – SubjectFull: Error Patterns
        Type: general
      – SubjectFull: Interrater Reliability
        Type: general
    Titles:
      – TitleFull: Seeing Is Solving: MLLMs, Reasoning, and Refusal in Visual Math
        Type: main
  BibRelationships:
    HasContributorRelationships:
      – PersonEntity:
          Name:
            NameFull: Ethan Croteau
      – PersonEntity:
          Name:
            NameFull: Neil Heffernan
    IsPartOfRelationships:
      – BibEntity:
          Dates:
            – D: 01
              M: 01
              Type: published
              Y: 2026
          Identifiers:
            – Type: issn-electronic
              Value: 2157-2100
          Numbering:
            – Type: volume
              Value: 18
            – Type: issue
              Value: 1
          Titles:
            – TitleFull: Journal of Educational Data Mining
              Type: main
ResultId 1