You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Anish Shrigondekar (Jira)" <ji...@apache.org> on 2025/08/01 00:27:00 UTC

[jira] [Closed] (SPARK-53001) Integrate RocksDB Memory Usage with the Unified Memory Manager

     [ https://issues.apache.org/jira/browse/SPARK-53001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Anish Shrigondekar closed SPARK-53001.
--------------------------------------

> Integrate RocksDB Memory Usage with the Unified Memory Manager
> --------------------------------------------------------------
>
>                 Key: SPARK-53001
>                 URL: https://issues.apache.org/jira/browse/SPARK-53001
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.1.0
>            Reporter: Eric Marnadi
>            Assignee: Eric Marnadi
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.1.0
>
>
> Currently, RocksDB memory is untracked and not included in memory decisions in Spark (particularly when Photon is enabled). We want to factor the RocksDB memory usage into memory allocations so we don't hit OOMs. This change introduces a background memory polling thread from the MemoryManager that queries RocksDB memory every X seconds (configurable via SQLConf).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org