Evaluating Large Language Models for Abstractive Summarization: A Benchmark Study on Recall, Fidelity, and Content Coherence.

Saved in:

Bibliographic Details
Title:	Evaluating Large Language Models for Abstractive Summarization: A Benchmark Study on Recall, Fidelity, and Content Coherence.
Authors:	SURABHI, ANURADHA¹ surabhiaim12023@gmail.com, MARTHA, SHESHIKALA¹
Source:	Journal of Information Science & Engineering. May2026, Vol. 42 Issue 3, p793-804. 12p.
Subjects:	Text summarization, Recall (Information retrieval), Natural language processing, Statistical accuracy, Language models, Benchmark problems (Computer science), Cohesion (Linguistics)
Abstract:	This study investigates the effectiveness of abstractive text summarization in the context of scientific documents using 40 diverse Large Language Models (LLMs). Unlike traditional extractive approaches that often produoe fragmented and loss coherent summaries, our work focuses on enhancing semantic fidelity, coherence, and comprehensive content coverage. Through a recall-oriented evaluation supported by BERT and METEOR metrics, our experimental results show that models such as Claude v2.1, Qwen-14B, Zephyr-7B, and Phi-3 emerged as top performers, achieving outstanding Fl scores above 0.93 and METEOR scores as high as 1.00. These models demonstrated a strong ability to retain critical information while producing fluent, human-like summaries. Our findings provide valuable benchmarks for selecting high-performing LLMs in summarization tasks and offer a foundation for future advancements, including domain adaptation, fact-checking integration, and multimodal summarization approaches in real-world Natural Language Processing applications. [ABSTRACT FROM AUTHOR]
	Copyright of Journal of Information Science & Engineering is the property of Institute of Information Science, Academia Sinica and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database:	Engineering Source

Description
Abstract:	This study investigates the effectiveness of abstractive text summarization in the context of scientific documents using 40 diverse Large Language Models (LLMs). Unlike traditional extractive approaches that often produoe fragmented and loss coherent summaries, our work focuses on enhancing semantic fidelity, coherence, and comprehensive content coverage. Through a recall-oriented evaluation supported by BERT and METEOR metrics, our experimental results show that models such as Claude v2.1, Qwen-14B, Zephyr-7B, and Phi-3 emerged as top performers, achieving outstanding Fl scores above 0.93 and METEOR scores as high as 1.00. These models demonstrated a strong ability to retain critical information while producing fluent, human-like summaries. Our findings provide valuable benchmarks for selecting high-performing LLMs in summarization tasks and offer a foundation for future advancements, including domain adaptation, fact-checking integration, and multimodal summarization approaches in real-world Natural Language Processing applications. [ABSTRACT FROM AUTHOR]
ISSN:	10162364
DOI:	10.6688/JISE.202605_42(3).0018