You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "kete1987 (via GitHub)" <gi...@apache.org> on 2026/05/31 14:32:44 UTC
[PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
kete1987 opened a new pull request, #56234:
URL: https://github.com/apache/spark/pull/56234
### What changes were proposed in this pull request?
In unbounded memory mode (the default, `boundedMemoryUsage = false`), `RocksDBMemoryManager` creates a new `LRUCache` per instance:
https://github.com/apache/spark/blob/d7df1920cfb8577c17a9b1555592d3772f061417/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBMemoryManager.scala#L161
but `RocksDB.close()` never calls `lruCache.close()`:
https://github.com/apache/spark/blob/d7df1920cfb8577c17a9b1555592d3772f061417/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala#L2614-L2627
The Java `LRUCache` wrapper holds a C++ `shared_ptr<Cache>`, so the native object is only freed when the JVM GC finalizes the wrapper — which rarely happens under low heap pressure. This causes native memory to accumulate until GC eventually runs, leading to OOM kills in long-running processes or CI runs with many RocksDB-heavy test suites.
The fix adds an explicit `lruCache.close()` call in `RocksDB.close()` for unbounded mode. In bounded mode the cache is a shared singleton managed by `RocksDBMemoryManager` and must not be closed per instance.
This is a separate issue from SPARK-56523 (Statistics native memory leak), which was already fixed.
### Why are the changes needed?
Without explicit `close()`, each `RocksDB` instance in unbounded mode leaks one `LRUCache` worth of native memory (`blockCacheSizeMB`, default 8 MB) for as long as GC does not run. The memory is never reclaimed deterministically.
A standalone reproducer tool confirms ~8.5 MB of native memory growth per open/close cycle in leak mode vs flat memory in fixed mode:
https://github.com/kete1987/rocksdb-leak-tool
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added a test in `RocksDBSuite` (`SPARK-57183: LRUCache is closed on RocksDB.close() in unbounded memory mode`) that verifies the native handle is released after `close()` via `LRUCache.isOwningHandle()`.
I affirm that the contribution is my original work and that I license the work to the project under the project's open source license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #56234:
URL: https://github.com/apache/spark/pull/56234#issuecomment-4612481661
@kete1987 branch-4.2 and branch-4.x are two different branches...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #56234:
URL: https://github.com/apache/spark/pull/56234#discussion_r3335553234
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -2133,6 +2133,15 @@ class RocksDB(
nativeStats.close()
rocksDbOptions.close()
dbLogger.close()
+ // In unbounded memory mode each RocksDB instance owns its LRUCache. Without explicit
+ // close() the native C++ cache object is only freed when the JVM GC finalizes the Java
+ // wrapper -- which rarely happens under low heap pressure. Closing explicitly here
+ // ensures native memory is reclaimed deterministically when the instance is released.
+ // In bounded mode the cache is a shared singleton managed by RocksDBMemoryManager
+ // and must not be closed here.
+ if (!conf.boundedMemoryUsage && lruCache != null) {
Review Comment:
So this is just releasing memory more proactively IIUC ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #56234:
URL: https://github.com/apache/spark/pull/56234#discussion_r3338541008
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -3276,6 +3276,36 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithColumnFamilies(
+ "SPARK-57183: LRUCache is closed on RocksDB.close() in unbounded memory mode",
+ TestWithBothChangelogCheckpointingEnabledAndDisabled) { colFamiliesEnabled =>
+ // In unbounded mode each RocksDB instance owns its LRUCache. Verify that close() releases
+ // the native handle so that the cache object is no longer valid after close.
+ withTempDir { dir =>
+ try {
+ val sqlConf = new SQLConf
+ sqlConf.setConfString(
+ RocksDBConf.ROCKSDB_SQL_CONF_NAME_PREFIX + "." +
+ RocksDBConf.BOUNDED_MEMORY_USAGE_CONF_KEY, "false")
Review Comment:
Oh can we combine the code into a single function - and just do a conditional check based on the value passed for `boundedMemoryUsageEnabled` ? that way - we don't need to duplicate the code and can mostly reuse it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #56234:
URL: https://github.com/apache/spark/pull/56234#issuecomment-4589443521
I'll ask other folks to review the change, but I'm going to give a general suggestion.
Please do not remove the section of PR template and consider filling the section as one of the requirement/duty.
> I affirm that the contribution is my original work and that I license the work to the project under the project's open source license.
This doesn't replace the requirement the PR template asks about the usage of LLM model and the clarification of the model. The template guides how to describe the model - please check it out again.
https://github.com/apache/spark/blob/master/.github/PULL_REQUEST_TEMPLATE
```
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #56234:
URL: https://github.com/apache/spark/pull/56234#issuecomment-4589447505
cc. @anishshri-db @micheal-o
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #56234:
URL: https://github.com/apache/spark/pull/56234#discussion_r3335577421
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -3276,6 +3276,36 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithColumnFamilies(
+ "SPARK-57183: LRUCache is closed on RocksDB.close() in unbounded memory mode",
+ TestWithBothChangelogCheckpointingEnabledAndDisabled) { colFamiliesEnabled =>
+ // In unbounded mode each RocksDB instance owns its LRUCache. Verify that close() releases
+ // the native handle so that the cache object is no longer valid after close.
+ withTempDir { dir =>
+ try {
+ val sqlConf = new SQLConf
+ sqlConf.setConfString(
+ RocksDBConf.ROCKSDB_SQL_CONF_NAME_PREFIX + "." +
+ RocksDBConf.BOUNDED_MEMORY_USAGE_CONF_KEY, "false")
Review Comment:
Can we add a test to verify that the bounded case is not affected by this change ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #56234:
URL: https://github.com/apache/spark/pull/56234#issuecomment-4608627598
why it's not merged in branch 4.x? this is not a breaking change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #56234:
URL: https://github.com/apache/spark/pull/56234#issuecomment-4610381488
Where are we? Do we not have a fix in 4.x? Shall we just cherry-pick and push as we would do it in merge script?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on code in PR #56234:
URL: https://github.com/apache/spark/pull/56234#discussion_r3337171275
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -2133,6 +2133,15 @@ class RocksDB(
nativeStats.close()
rocksDbOptions.close()
dbLogger.close()
+ // In unbounded memory mode each RocksDB instance owns its LRUCache. Without explicit
+ // close() the native C++ cache object is only freed when the JVM GC finalizes the Java
+ // wrapper -- which rarely happens under low heap pressure. Closing explicitly here
+ // ensures native memory is reclaimed deterministically when the instance is released.
+ // In bounded mode the cache is a shared singleton managed by RocksDBMemoryManager
+ // and must not be closed here.
+ if (!conf.boundedMemoryUsage && lruCache != null) {
Review Comment:
Yes, exactly. In unbounded mode each RocksDB instance owns its LRUCache exclusively, so it's correct to close it deterministically on close(). Without this, the native C++ object is only freed when the JVM GC finalizes the Java wrapper, which is non-deterministic and can be significantly delayed under low heap pressure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on code in PR #56234:
URL: https://github.com/apache/spark/pull/56234#discussion_r3339179315
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -3276,6 +3276,36 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithColumnFamilies(
+ "SPARK-57183: LRUCache is closed on RocksDB.close() in unbounded memory mode",
+ TestWithBothChangelogCheckpointingEnabledAndDisabled) { colFamiliesEnabled =>
+ // In unbounded mode each RocksDB instance owns its LRUCache. Verify that close() releases
+ // the native handle so that the cache object is no longer valid after close.
+ withTempDir { dir =>
+ try {
+ val sqlConf = new SQLConf
+ sqlConf.setConfString(
+ RocksDBConf.ROCKSDB_SQL_CONF_NAME_PREFIX + "." +
+ RocksDBConf.BOUNDED_MEMORY_USAGE_CONF_KEY, "false")
Review Comment:
Good suggestion, combined both cases into a single parameterized test using `Seq(true, false).foreach`, covering bounded and unbounded modes without code duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on PR #56234:
URL: https://github.com/apache/spark/pull/56234#issuecomment-4616108224
@cloud-fan opened #56308 for branch-4.x. Sorry for the confusion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #56234:
URL: https://github.com/apache/spark/pull/56234#issuecomment-4608629184
@kete1987 can you open a 4.x PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on PR #56234:
URL: https://github.com/apache/spark/pull/56234#issuecomment-4610953308
@HeartSaVioR @cloud-fan Already done — #56281 was merged to branch-4.2. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on code in PR #56234:
URL: https://github.com/apache/spark/pull/56234#discussion_r3337177193
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -3276,6 +3276,36 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithColumnFamilies(
+ "SPARK-57183: LRUCache is closed on RocksDB.close() in unbounded memory mode",
+ TestWithBothChangelogCheckpointingEnabledAndDisabled) { colFamiliesEnabled =>
+ // In unbounded mode each RocksDB instance owns its LRUCache. Verify that close() releases
+ // the native handle so that the cache object is no longer valid after close.
+ withTempDir { dir =>
+ try {
+ val sqlConf = new SQLConf
+ sqlConf.setConfString(
+ RocksDBConf.ROCKSDB_SQL_CONF_NAME_PREFIX + "." +
+ RocksDBConf.BOUNDED_MEMORY_USAGE_CONF_KEY, "false")
Review Comment:
Good point, added a test for the bounded case that verifies the shared LRUCache handle remains open after a single RocksDB instance closes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "kete1987 (via GitHub)" <gi...@apache.org>.
kete1987 commented on PR #56234:
URL: https://github.com/apache/spark/pull/56234#issuecomment-4590231154
@HeartSaVioR Thanks for the feedback! I've updated the PR description to include the AI tooling section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db closed pull request #56234: [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode
URL: https://github.com/apache/spark/pull/56234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57183][SS] Close LRUCache on RocksDB.close() in unbounded memory mode [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #56234:
URL: https://github.com/apache/spark/pull/56234#discussion_r3338541008
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -3276,6 +3276,36 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithColumnFamilies(
+ "SPARK-57183: LRUCache is closed on RocksDB.close() in unbounded memory mode",
+ TestWithBothChangelogCheckpointingEnabledAndDisabled) { colFamiliesEnabled =>
+ // In unbounded mode each RocksDB instance owns its LRUCache. Verify that close() releases
+ // the native handle so that the cache object is no longer valid after close.
+ withTempDir { dir =>
+ try {
+ val sqlConf = new SQLConf
+ sqlConf.setConfString(
+ RocksDBConf.ROCKSDB_SQL_CONF_NAME_PREFIX + "." +
+ RocksDBConf.BOUNDED_MEMORY_USAGE_CONF_KEY, "false")
Review Comment:
Oh can we combine the code into a single function/test - and just do a conditional check based on the value passed for `boundedMemoryUsageEnabled` ? that way - we don't need to duplicate the code and can mostly reuse it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org