(spark) branch master updated: [SPARK-51675][SS][FOLLOW-UP] Clear internal maps on close to remove DB open dep in other places

[ka...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

kabhwan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 3ab0f2de54af [SPARK-51675][SS][FOLLOW-UP] Clear internal maps on close to remove DB open dep in other places
3ab0f2de54af is described below

commit 3ab0f2de54af19daf09450ccad4fe79dac075cab
Author: Anish Shrigondekar <an...@databricks.com>
AuthorDate: Fri Apr 4 10:41:17 2025 +0900

    [SPARK-51675][SS][FOLLOW-UP] Clear internal maps on close to remove DB open dep in other places
    
    ### What changes were proposed in this pull request?
    Clear internal maps on close to remove DB open dep in other places
    
    ### Why are the changes needed?
    Refactor related change
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Verified that tests failed without db open check and not clearing the internal state.
    
    ```
    ===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.streaming.state.RocksDBSuite, threads: rpc-boss-3-1 (daemon=true), ForkJoinPool.commonPool-worker-2 (daemon=true), shuffle-boss-6-1 (daemon=true), ForkJoinPool.commonPool-worker-1 (daemon=true) =====
    [info] Run completed in 11 seconds.
    [info] Total number of tests run: 16
    [info] Suites: completed 1, aborted 0
    [info] Tests: succeeded 16, failed 0, canceled 0, ignored 0, pending 0
    [info] All tests passed.
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #50510 from anishshri-db/task/SPARK-51715.
    
    Authored-by: Anish Shrigondekar <an...@databricks.com>
    Signed-off-by: Jungtaek Lim <ka...@gmail.com>
---
 .../org/apache/spark/sql/execution/streaming/state/RocksDB.scala      | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
index b324dc73f529..0fbb2c0ff367 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
@@ -267,7 +267,7 @@ class RocksDB(
    * @return - true if the column family exists, false otherwise
    */
   def checkColFamilyExists(colFamilyName: String): Boolean = {
-    db != null && colFamilyNameToInfoMap.containsKey(colFamilyName)
+    colFamilyNameToInfoMap.containsKey(colFamilyName)
   }
 
   // This method sets the internal column family metadata to
@@ -1215,6 +1215,8 @@ class RocksDB(
       }
 
       silentDeleteRecursively(localRootDir, "closing RocksDB")
+      // Clear internal maps to reset the state
+      clearColFamilyMaps()
     } catch {
       case e: Exception =>
         logWarning("Error closing RocksDB", e)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org