You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Eric Marnadi (Jira)" <ji...@apache.org> on 2025/08/01 04:53:00 UTC

[jira] [Updated] (SPARK-53001) Integrate RocksDB Memory Usage with the Unified Memory Manager

     [ https://issues.apache.org/jira/browse/SPARK-53001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Marnadi updated SPARK-53001:
---------------------------------
    Description: Currently, RocksDB memory is untracked and not included in memory decisions in Spark. We want to factor the RocksDB memory usage into memory allocations so we don't hit OOMs. This change introduces a background memory polling thread from the MemoryManager that queries RocksDB memory every X seconds (configurable via SQLConf).  (was: Currently, RocksDB memory is untracked and not included in memory decisions in Spark (particularly when Photon is enabled). We want to factor the RocksDB memory usage into memory allocations so we don't hit OOMs. This change introduces a background memory polling thread from the MemoryManager that queries RocksDB memory every X seconds (configurable via SQLConf).)

> Integrate RocksDB Memory Usage with the Unified Memory Manager
> --------------------------------------------------------------
>
>                 Key: SPARK-53001
>                 URL: https://issues.apache.org/jira/browse/SPARK-53001
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.1.0
>            Reporter: Eric Marnadi
>            Assignee: Eric Marnadi
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> Currently, RocksDB memory is untracked and not included in memory decisions in Spark. We want to factor the RocksDB memory usage into memory allocations so we don't hit OOMs. This change introduces a background memory polling thread from the MemoryManager that queries RocksDB memory every X seconds (configurable via SQLConf).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org