A Computational Framework to Study Hierarchical Processing in Visual Narratives.
Saved in:
| Title: | A Computational Framework to Study Hierarchical Processing in Visual Narratives. |
|---|---|
| Authors: | Upadhyayula, Aditya1 (AUTHOR) aditya.usa8@gmail.com, Cohn, Neil2 (AUTHOR) |
| Source: | Cognitive Science. May2025, Vol. 49 Issue 5, p1-29. 29p. |
| Subject Terms: | *Cognitive structures, Hidden Markov models, Language models, Markov processes, Mental representation, Psycholinguistics |
| Abstract: | Theories of visual narrative comprehension have advocated for a hierarchical grammar‐based comprehension mechanism, but only limited work has investigated this hierarchy. Here, we provide a computational framework inspired by computational psycholinguistics to address hierarchy in visual narratives. The predictions generated by this framework were compared against behavior data to draw inferences about the hierarchical properties of visual narratives. A segmentation task—where participants ranked all possible segmental boundaries—demonstrated that participants' preferences were predicted by visual narrative grammar. Three kinds of models using surprisal theory—an Earley parser, a hidden Markov model (HMM), and an n‐gram model—were then used to generate segmentation preferences for the same task. Earley parser's preferences were based on a hierarchical grammar with recursion properties, while the HMM and the n‐grams used a flattened grammar for visual narrative comprehension. Given the differences in the mechanics of these models, contrasting their predictions against behavior data could provide crucial insights into understanding the underlying mechanisms of visual narrative comprehension. By investigating grammatical systems outside of language, this research provides new directions to explore the generic makeup of the cognitive structure of mental representations. [ABSTRACT FROM AUTHOR] |
| Copyright of Cognitive Science is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Database: | Education Research Complete |
|
Full text is not displayed to guests.
Login for full access.
|
|
| Abstract: | Theories of visual narrative comprehension have advocated for a hierarchical grammar‐based comprehension mechanism, but only limited work has investigated this hierarchy. Here, we provide a computational framework inspired by computational psycholinguistics to address hierarchy in visual narratives. The predictions generated by this framework were compared against behavior data to draw inferences about the hierarchical properties of visual narratives. A segmentation task—where participants ranked all possible segmental boundaries—demonstrated that participants' preferences were predicted by visual narrative grammar. Three kinds of models using surprisal theory—an Earley parser, a hidden Markov model (HMM), and an n‐gram model—were then used to generate segmentation preferences for the same task. Earley parser's preferences were based on a hierarchical grammar with recursion properties, while the HMM and the n‐grams used a flattened grammar for visual narrative comprehension. Given the differences in the mechanics of these models, contrasting their predictions against behavior data could provide crucial insights into understanding the underlying mechanisms of visual narrative comprehension. By investigating grammatical systems outside of language, this research provides new directions to explore the generic makeup of the cognitive structure of mental representations. [ABSTRACT FROM AUTHOR] |
|---|---|
| ISSN: | 03640213 |
| DOI: | 10.1111/cogs.70050 |
