A Computational Framework to Study Hierarchical Processing in Visual Narratives.
Saved in:
| Title: | A Computational Framework to Study Hierarchical Processing in Visual Narratives. |
|---|---|
| Authors: | Upadhyayula, Aditya1 (AUTHOR) aditya.usa8@gmail.com, Cohn, Neil2 (AUTHOR) |
| Source: | Cognitive Science. May2025, Vol. 49 Issue 5, p1-29. 29p. |
| Subject Terms: | *Cognitive structures, Hidden Markov models, Language models, Markov processes, Mental representation, Psycholinguistics |
| Abstract: | Theories of visual narrative comprehension have advocated for a hierarchical grammar‐based comprehension mechanism, but only limited work has investigated this hierarchy. Here, we provide a computational framework inspired by computational psycholinguistics to address hierarchy in visual narratives. The predictions generated by this framework were compared against behavior data to draw inferences about the hierarchical properties of visual narratives. A segmentation task—where participants ranked all possible segmental boundaries—demonstrated that participants' preferences were predicted by visual narrative grammar. Three kinds of models using surprisal theory—an Earley parser, a hidden Markov model (HMM), and an n‐gram model—were then used to generate segmentation preferences for the same task. Earley parser's preferences were based on a hierarchical grammar with recursion properties, while the HMM and the n‐grams used a flattened grammar for visual narrative comprehension. Given the differences in the mechanics of these models, contrasting their predictions against behavior data could provide crucial insights into understanding the underlying mechanisms of visual narrative comprehension. By investigating grammatical systems outside of language, this research provides new directions to explore the generic makeup of the cognitive structure of mental representations. [ABSTRACT FROM AUTHOR] |
| Copyright of Cognitive Science is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
|
Full text is not displayed to guests.
Login for full access.
|
|
| FullText | Links: – Type: pdflink Text: Availability: 1 |
|---|---|
| Header | DbId: ehh DbLabel: Education Research Complete An: 185399543 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 0 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: A Computational Framework to Study Hierarchical Processing in Visual Narratives. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Upadhyayula%2C+Aditya%22">Upadhyayula, Aditya</searchLink><relatesTo>1</relatesTo> (AUTHOR)<i> aditya.usa8@gmail.com</i><br /><searchLink fieldCode="AR" term="%22Cohn%2C+Neil%22">Cohn, Neil</searchLink><relatesTo>2</relatesTo> (AUTHOR) – Name: TitleSource Label: Source Group: Src Data: <searchLink fieldCode="JN" term="%22Cognitive+Science%22">Cognitive Science</searchLink>. May2025, Vol. 49 Issue 5, p1-29. 29p. – Name: Subject Label: Subject Terms Group: Su Data: *<searchLink fieldCode="DE" term="%22Cognitive+structures%22">Cognitive structures</searchLink><br /><searchLink fieldCode="DE" term="%22Hidden+Markov+models%22">Hidden Markov models</searchLink><br /><searchLink fieldCode="DE" term="%22Language+models%22">Language models</searchLink><br /><searchLink fieldCode="DE" term="%22Markov+processes%22">Markov processes</searchLink><br /><searchLink fieldCode="DE" term="%22Mental+representation%22">Mental representation</searchLink><br /><searchLink fieldCode="DE" term="%22Psycholinguistics%22">Psycholinguistics</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: Theories of visual narrative comprehension have advocated for a hierarchical grammar‐based comprehension mechanism, but only limited work has investigated this hierarchy. Here, we provide a computational framework inspired by computational psycholinguistics to address hierarchy in visual narratives. The predictions generated by this framework were compared against behavior data to draw inferences about the hierarchical properties of visual narratives. A segmentation task—where participants ranked all possible segmental boundaries—demonstrated that participants' preferences were predicted by visual narrative grammar. Three kinds of models using surprisal theory—an Earley parser, a hidden Markov model (HMM), and an n‐gram model—were then used to generate segmentation preferences for the same task. Earley parser's preferences were based on a hierarchical grammar with recursion properties, while the HMM and the n‐grams used a flattened grammar for visual narrative comprehension. Given the differences in the mechanics of these models, contrasting their predictions against behavior data could provide crucial insights into understanding the underlying mechanisms of visual narrative comprehension. By investigating grammatical systems outside of language, this research provides new directions to explore the generic makeup of the cognitive structure of mental representations. [ABSTRACT FROM AUTHOR] – Name: AbstractSuppliedCopyright Label: Group: Ab Data: <i>Copyright of Cognitive Science is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=ehh&AN=185399543 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.1111/cogs.70050 Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 29 StartPage: 1 Subjects: – SubjectFull: Cognitive structures Type: general – SubjectFull: Hidden Markov models Type: general – SubjectFull: Language models Type: general – SubjectFull: Markov processes Type: general – SubjectFull: Mental representation Type: general – SubjectFull: Psycholinguistics Type: general Titles: – TitleFull: A Computational Framework to Study Hierarchical Processing in Visual Narratives. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Upadhyayula, Aditya – PersonEntity: Name: NameFull: Cohn, Neil IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 05 Text: May2025 Type: published Y: 2025 Identifiers: – Type: issn-print Value: 03640213 Numbering: – Type: volume Value: 49 – Type: issue Value: 5 Titles: – TitleFull: Cognitive Science Type: main |
| ResultId | 1 |
