GC-Risk-Aware Adaptive Hierarchical Persistence Strategy for Apache Spark.

Saved in:

Bibliographic Details
Title:	GC-Risk-Aware Adaptive Hierarchical Persistence Strategy for Apache Spark.
Authors:	Ren, Yufei¹ ffsdw@henu.edu.cn, Fei, Haifeng¹ fhf@henu.edu.cn, Du, Weipeng¹ wpdu@henu.edu.cn, Wang, Congyang¹ wangcongyang@henu.edu.cn, Yuan, Hang¹ yuanhang@henu.edu.cn, Yu, Junyang² jyyu@henu.edu.cn
Source:	Engineering Letters. Oct2026, Vol. 34 Issue 10, p4254-4267. 14p.
Subjects:	Computer memory management
Abstract:	Apache Spark's Unified Memory Management (UMM) model improves memory utilization by allowing execution memory and storage memory to share heap space. Under memory pressure, however, this sharing design creates asymmetric contention. Execution tasks can evict cached Resilient Distributed Dataset (RDD) blocks, whereas cached data cannot reclaim memory occupied by execution tasks. This asymmetry can reduce cache hit rates, increase garbage collection (GC) overhead, and trigger out-of-memory (OOM) failures. To address these problems, this paper proposes GAPS, a GCrisk-aware adaptive hierarchical persistence strategy for Spark. GAPS uses a two-stage process to make persistence decisions. First, a Cache Priority Score (CPS) model combines static directed acyclic graph (DAG) analysis with runtime metrics. It estimates the caching value of each RDD from temporal reuse, topological influence, and recomputation cost. Low-benefit data are filtered out before placement is considered. Second, a Persistence Location Selection (PLS) model formulates placement as a hierarchical net-utility problem. It selects on-heap memory, off-heap memory, or disk according to the predicted benefit and storage cost of each candidate tier. A sigmoid-based GC-risk function penalizes on-heap placement when heap utilization enters a high-risk region. This penalty allows GAPS to shift data away from the JVM heap before GC pressure becomes severe. Experiments on PageRank, Logistic Regression, TeraSort, and WordCount show that GAPS improves cache stability and reduces GC overhead under memory-constrained workloads. Compared with native Spark persistence strategies, including MEMORY ONLY, MEMORY AND DISK, OFF HEAP, and DISK ONLY, GAPS reduces job execution time by up to 55.6%. [ABSTRACT FROM AUTHOR]
	Copyright of Engineering Letters is the property of International Association of Engineers (IAENG) and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database:	Engineering Source

Description
Abstract:	Apache Spark's Unified Memory Management (UMM) model improves memory utilization by allowing execution memory and storage memory to share heap space. Under memory pressure, however, this sharing design creates asymmetric contention. Execution tasks can evict cached Resilient Distributed Dataset (RDD) blocks, whereas cached data cannot reclaim memory occupied by execution tasks. This asymmetry can reduce cache hit rates, increase garbage collection (GC) overhead, and trigger out-of-memory (OOM) failures. To address these problems, this paper proposes GAPS, a GCrisk-aware adaptive hierarchical persistence strategy for Spark. GAPS uses a two-stage process to make persistence decisions. First, a Cache Priority Score (CPS) model combines static directed acyclic graph (DAG) analysis with runtime metrics. It estimates the caching value of each RDD from temporal reuse, topological influence, and recomputation cost. Low-benefit data are filtered out before placement is considered. Second, a Persistence Location Selection (PLS) model formulates placement as a hierarchical net-utility problem. It selects on-heap memory, off-heap memory, or disk according to the predicted benefit and storage cost of each candidate tier. A sigmoid-based GC-risk function penalizes on-heap placement when heap utilization enters a high-risk region. This penalty allows GAPS to shift data away from the JVM heap before GC pressure becomes severe. Experiments on PageRank, Logistic Regression, TeraSort, and WordCount show that GAPS improves cache stability and reduces GC overhead under memory-constrained workloads. Compared with native Spark persistence strategies, including MEMORY ONLY, MEMORY AND DISK, OFF HEAP, and DISK ONLY, GAPS reduces job execution time by up to 55.6%. [ABSTRACT FROM AUTHOR]
ISSN:	1816093X