[jira] [Created] (HUDI-9199) OOM when querying MOR table due to redundant copy of deltalog SpillableMap into memory

["Moran (Jira)" <ji...@apache.org>]

Moran created HUDI-9199:
---------------------------

             Summary: OOM when querying MOR table due to redundant copy of deltalog SpillableMap into memory
                 Key: HUDI-9199
                 URL: https://issues.apache.org/jira/browse/HUDI-9199
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Moran


When querying a Merge-On-Read (MOR) Hudi table, the Spark executor encounters an Out-of-Memory (OOM) error. Upon investigation, it was observed that the root cause lies in the handling of deltalog files during the read path. Specifically:
 # {*}Redundant Copy Operation{*}:
After reading deltalog entries into a {{SpillableMap}} (which is designed to spill data to disk when memory thresholds are exceeded), an additional unnecessary *in-memory copy* of the data is performed. This defeats the purpose of {{SpillableMap}} and forces all data to reside in memory instead of spilling to disk as intended.

 # {*}Key Code Path{*}:
https://github.com/apache/hudi/blob/8b7bd1391bd61600d23b8defed4cdc7d789502d1/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/Iterators.scala#L416
 # {*}Impact{*}:
For large deltalog files or tables with frequent updates, this redundant copy operation leads to excessive memory consumption, ultimately causing OOM errors and query failures.

 # {*}Expected Behavior{*}:
The {{SpillableMap}} should manage memory/disk spilling transparently, and no unnecessary in-memory copies should occur during the deltalog processing phase.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)