You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "liviazhu-db (via GitHub)" <gi...@apache.org> on 2024/10/29 21:59:18 UTC

[PR] [SPARK-50163] Fix the RocksDB extra release due to the completion listener [spark]

liviazhu-db opened a new pull request, #48697:
URL: https://github.com/apache/spark/pull/48697

   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error type or message, please read the guideline first in
        'common/utils/src/main/resources/error/README.md'.
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Adds a new `releaseForThread(opType, threadRef)` method that only releases the RocksDB acquireLock if the provided threadRef is the same as the threadRef in `acquiredThreadInfo` and modify the task completion listener created in `acquire` to use the new method so that it doesn't accidentally release the lock if it's owned by another thread. 
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   There currently exists a race condition in RocksDB where the completion listener may release a lock validly held by another thread, resulting in a thread-unsafe condition. We have seen this bug in production.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   No
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
   -->
   Added unit test in RocksDB.scala, all other unit tests continue to pass as expected.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   <!--
   If generative AI tooling has been used in the process of authoring this patch, please include the
   phrase: 'Generated-by: ' followed by the name of the tool and its version.
   If no, write 'No'.
   Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
   -->
   Generated-by: Github Co-pilot 1.5.25.10


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SS] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1824900931


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -787,9 +790,9 @@ class RocksDB(
 
   /** Release all resources */
   def close(): Unit = {
+    // Acquire DB instance lock and release at the end to allow for synchronized access
+    acquire(CloseStore)

Review Comment:
   Good point. I can wrap it in another try block that swallows the exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1826170578


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,39 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {

Review Comment:
   Changed it to use pattern matching



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1826335482


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.

Review Comment:
   Same as above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823529717


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -787,9 +790,9 @@ class RocksDB(
 
   /** Release all resources */
   def close(): Unit = {
+    // Acquire DB instance lock and release at the end to allow for synchronized access
+    acquire(CloseStore)

Review Comment:
   Previously we swallowed any exception that happens here (see the catch). By moving this `acquire()` call out, if we fail to acquire lock, which then throws exception, we will not swallow it. Is that the intention here?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -903,8 +906,8 @@ class RocksDB(
    */
   def metricsOpt: Option[RocksDBMetrics] = {
     var rocksDBMetricsOpt: Option[RocksDBMetrics] = None
+    acquire(ReportStoreMetrics)
     try {

Review Comment:
   ditto: lock acquisition exception will not be swallowed here by moving this out



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && acquiredThreadInfo != releaseForThreadOpt.get) {
+          logInfo(log"Thread info does not match the acquired thread when attempting to" +

Review Comment:
   nit: also add the releaseThreadInfo?



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,166 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // Create a custom ExecutionContext with 3 threads
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonFixedThreadPool(5, "pool-thread-executor"))
+    val timeout = 5.seconds
+
+    val stateLock = new Object()
+    var state = 0
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |
+    // 1.    |                  | task complete    | wait for s3      |
+    //       |                  | signal s2, END   |                  |
+    // 2.    | start thread 3   |                  |                  | load, commit
+    //       | wait for timeout |                  |                  | timeout, END
+    //       | signal s3        |                  |                  |
+    // 3.    | wait for s4      |                  | commit           |
+    //       |                  |                  | signal s4, END   |
+    // 4.    | close db, END    |                  |                  |
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      val conf = dbConf.copy(lockAcquireTimeoutMs = timeout.toMillis.toInt)
+      val db = new RocksDB(
+        remoteDir,
+        conf = conf,
+        localRootDir = Utils.createTempDir(),
+        hadoopConf = new Configuration(),
+        loggingId = s"[Thread-${Thread.currentThread.getId}]",
+        useColumnFamilies = false
+      )
+      try {
+        Future { // THREAD 1
+          // Set thread 1's task context so that it is not a clone
+          // of withDB's taskContext, which will close the db when the
+          // task is marked as complete
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          // Simulate a task that loads and commits, db should be unlocked after
+          db.load(0)
+          db.put("a", "1")
+          db.commit()
+
+          Future { // THREAD 2
+            // Set thread 2's task context so that it is not a clone of thread 1's
+            // so it won't be marked as complete
+            val taskContext = TaskContext.empty()
+            TaskContext.setTaskContext(taskContext)
+
+            // Load the db and signal that we have entered state 1
+            stateLock.synchronized {
+              db.load(1)
+              state = 1
+              stateLock.notifyAll()
+            }
+
+            stateLock.synchronized {
+              // Wait until we have entered state 3, commit and signal
+              // that we have entered state 4
+              while (state != 3) {
+                stateLock.wait()
+              }
+              db.commit()
+              state = 4
+              stateLock.notifyAll()
+            }
+          }
+
+          stateLock.synchronized {
+            // Wait until we have entered state 1

Review Comment:
   I think we need to improve the comments in this test. And explain why we are waiting and what we are testing. Example: here we are waiting because we want to let thread 2 to have loaded the db (hence acquired lock), so that we can test that running task completion for  thread 1, will not release the lock acquired by 2.



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && acquiredThreadInfo != releaseForThreadOpt.get) {

Review Comment:
   Question: in `acquire()` method, we compare by comparing the threadId, i.e.
   `newAcquiredThreadInfo.threadRef.get.get.getId != acquiredThreadInfo.threadRef.get.get.getId`
   
   But here we are comparing using the entire `ThreadInfo`. Trying to understand why? Why are we not doing this in `acquire`? Is there a reason? Ideally, we should use a consistent approach. cc @anishshri-db 



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,166 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // Create a custom ExecutionContext with 3 threads

Review Comment:
   nit: is num of threads 3 or 5?



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,166 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // Create a custom ExecutionContext with 3 threads
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonFixedThreadPool(5, "pool-thread-executor"))
+    val timeout = 5.seconds
+
+    val stateLock = new Object()
+    var state = 0
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |
+    // 1.    |                  | task complete    | wait for s3      |
+    //       |                  | signal s2, END   |                  |
+    // 2.    | start thread 3   |                  |                  | load, commit
+    //       | wait for timeout |                  |                  | timeout, END
+    //       | signal s3        |                  |                  |
+    // 3.    | wait for s4      |                  | commit           |
+    //       |                  |                  | signal s4, END   |
+    // 4.    | close db, END    |                  |                  |
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      val conf = dbConf.copy(lockAcquireTimeoutMs = timeout.toMillis.toInt)
+      val db = new RocksDB(
+        remoteDir,
+        conf = conf,
+        localRootDir = Utils.createTempDir(),
+        hadoopConf = new Configuration(),
+        loggingId = s"[Thread-${Thread.currentThread.getId}]",
+        useColumnFamilies = false
+      )
+      try {
+        Future { // THREAD 1
+          // Set thread 1's task context so that it is not a clone
+          // of withDB's taskContext, which will close the db when the
+          // task is marked as complete
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          // Simulate a task that loads and commits, db should be unlocked after
+          db.load(0)
+          db.put("a", "1")
+          db.commit()
+
+          Future { // THREAD 2
+            // Set thread 2's task context so that it is not a clone of thread 1's
+            // so it won't be marked as complete
+            val taskContext = TaskContext.empty()
+            TaskContext.setTaskContext(taskContext)
+
+            // Load the db and signal that we have entered state 1
+            stateLock.synchronized {
+              db.load(1)
+              state = 1
+              stateLock.notifyAll()
+            }
+
+            stateLock.synchronized {
+              // Wait until we have entered state 3, commit and signal
+              // that we have entered state 4
+              while (state != 3) {
+                stateLock.wait()
+              }

Review Comment:
   I don't see assertions in this test, to confirm our assumptions. Do we need to assert here that the current acquiredThreadInfo is actually for this thread? To confirm that it wasn't released. We should do that in other places needed to.
   
   Because for example in this place, if the lock has been accidentally released, commit() will still work since we don't acquire lock in commit and just release the lock acquired in load.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "siying (via GitHub)" <gi...@apache.org>.
siying commented on PR #48697:
URL: https://github.com/apache/spark/pull/48697#issuecomment-2450418693

   @liviazhu-db it makes sense. It's good to go now. Perhaps we should take note for a future code refactoring. Maybe we introduce some concept like session so we have this information. It would also have other benefits.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823205970


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -743,17 +743,21 @@ class RocksDB(
    */
   def rollback(): Unit = {
     acquire(RollbackStore)
-    numKeysOnWritingVersion = numKeysOnLoadedVersion
-    loadedVersion = -1L
-    lastCommitBasedStateStoreCkptId = None
-    lastCommittedStateStoreCkptId = None
-    loadedStateStoreCkptId = None
-    sessionStateStoreCkptId = None
-    changelogWriter.foreach(_.abort())
-    // Make sure changelogWriter gets recreated next time.
-    changelogWriter = None
-    release(RollbackStore)
-    logInfo(log"Rolled back to ${MDC(LogKeys.VERSION_NUM, loadedVersion)}")
+    try {
+      numKeysOnWritingVersion = numKeysOnLoadedVersion
+      loadedVersion = -1L
+      lastCommitBasedStateStoreCkptId = None
+      lastCommittedStateStoreCkptId = None
+      loadedStateStoreCkptId = None
+      sessionStateStoreCkptId = None
+      changelogWriter.foreach(_.abort())
+      // Make sure changelogWriter gets recreated next time.
+      changelogWriter = None
+

Review Comment:
   Oh sorry I see, removed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823109535


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -956,13 +956,43 @@ class RocksDB(
       acquiredThreadInfo = newAcquiredThreadInfo
       // Add a listener to always release the lock when the task (if active) completes
       Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
-        _ => this.release(StoreTaskCompletionListener)
+        _ => this.releaseForThread(StoreTaskCompletionListener, newAcquiredThreadInfo.threadRef)

Review Comment:
   Discussed offline with @liviazhu-db 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823180364


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -956,51 +960,43 @@ class RocksDB(
       acquiredThreadInfo = newAcquiredThreadInfo
       // Add a listener to always release the lock when the task (if active) completes
       Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
-        _ => this.releaseForThread(StoreTaskCompletionListener, newAcquiredThreadInfo.threadRef)
+        _ => this.release(StoreTaskCompletionListener, Some(newAcquiredThreadInfo))
       })
       logInfo(log"RocksDB instance was acquired by ${MDC(LogKeys.THREAD, acquiredThreadInfo)} " +
         log"for opType=${MDC(LogKeys.OP_TYPE, opType.toString)}")
     }
   }
 
   /**
-   * Function to release RocksDB instance lock for if the provided thread is the one
-   * that acquired it.
+   * Function to release RocksDB instance lock that allows for synchronized access to the state
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
-   * @param threadRef - thread reference to check against the acquired thread
+   * @param forThreadOpt - optional thread to check against acquired thread
    */
-  private def releaseForThread(opType: RocksDBOpType,
-    threadRef: WeakReference[Thread]): Unit = acquireLock.synchronized {
-    if (acquiredThreadInfo != null && acquiredThreadInfo.threadRef.get.isDefined) {
-      if (!threadRef.get.isDefined) {
-        logInfo(log"Thread reference is empty when attempting to release for" +
-          log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
-          log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
-        return
-      }
+  private def release(opType: RocksDBOpType,

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1825047603


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && acquiredThreadInfo != releaseForThreadOpt.get) {

Review Comment:
   Reverted to just compare ID to be consistent here. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1824914794


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && acquiredThreadInfo != releaseForThreadOpt.get) {
+          logInfo(log"Thread info does not match the acquired thread when attempting to" +

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1825131414


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,166 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // Create a custom ExecutionContext with 3 threads

Review Comment:
   It's 3, fixed! 



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,166 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // Create a custom ExecutionContext with 3 threads
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonFixedThreadPool(5, "pool-thread-executor"))
+    val timeout = 5.seconds
+
+    val stateLock = new Object()
+    var state = 0
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |
+    // 1.    |                  | task complete    | wait for s3      |
+    //       |                  | signal s2, END   |                  |
+    // 2.    | start thread 3   |                  |                  | load, commit
+    //       | wait for timeout |                  |                  | timeout, END
+    //       | signal s3        |                  |                  |
+    // 3.    | wait for s4      |                  | commit           |
+    //       |                  |                  | signal s4, END   |
+    // 4.    | close db, END    |                  |                  |
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      val conf = dbConf.copy(lockAcquireTimeoutMs = timeout.toMillis.toInt)
+      val db = new RocksDB(
+        remoteDir,
+        conf = conf,
+        localRootDir = Utils.createTempDir(),
+        hadoopConf = new Configuration(),
+        loggingId = s"[Thread-${Thread.currentThread.getId}]",
+        useColumnFamilies = false
+      )
+      try {
+        Future { // THREAD 1
+          // Set thread 1's task context so that it is not a clone
+          // of withDB's taskContext, which will close the db when the
+          // task is marked as complete
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          // Simulate a task that loads and commits, db should be unlocked after
+          db.load(0)
+          db.put("a", "1")
+          db.commit()
+
+          Future { // THREAD 2
+            // Set thread 2's task context so that it is not a clone of thread 1's
+            // so it won't be marked as complete
+            val taskContext = TaskContext.empty()
+            TaskContext.setTaskContext(taskContext)
+
+            // Load the db and signal that we have entered state 1
+            stateLock.synchronized {
+              db.load(1)
+              state = 1
+              stateLock.notifyAll()
+            }
+
+            stateLock.synchronized {
+              // Wait until we have entered state 3, commit and signal
+              // that we have entered state 4
+              while (state != 3) {
+                stateLock.wait()
+              }

Review Comment:
   The main assertion of this test is that thread 3 will timeout, but I have also added additional assertions to ensure that the lock is held by the correct thread.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1825131706


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,166 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // Create a custom ExecutionContext with 3 threads
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonFixedThreadPool(5, "pool-thread-executor"))
+    val timeout = 5.seconds
+
+    val stateLock = new Object()
+    var state = 0
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |
+    // 1.    |                  | task complete    | wait for s3      |
+    //       |                  | signal s2, END   |                  |
+    // 2.    | start thread 3   |                  |                  | load, commit
+    //       | wait for timeout |                  |                  | timeout, END
+    //       | signal s3        |                  |                  |
+    // 3.    | wait for s4      |                  | commit           |
+    //       |                  |                  | signal s4, END   |
+    // 4.    | close db, END    |                  |                  |
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      val conf = dbConf.copy(lockAcquireTimeoutMs = timeout.toMillis.toInt)
+      val db = new RocksDB(
+        remoteDir,
+        conf = conf,
+        localRootDir = Utils.createTempDir(),
+        hadoopConf = new Configuration(),
+        loggingId = s"[Thread-${Thread.currentThread.getId}]",
+        useColumnFamilies = false
+      )
+      try {
+        Future { // THREAD 1
+          // Set thread 1's task context so that it is not a clone
+          // of withDB's taskContext, which will close the db when the
+          // task is marked as complete
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          // Simulate a task that loads and commits, db should be unlocked after
+          db.load(0)
+          db.put("a", "1")
+          db.commit()
+
+          Future { // THREAD 2
+            // Set thread 2's task context so that it is not a clone of thread 1's
+            // so it won't be marked as complete
+            val taskContext = TaskContext.empty()
+            TaskContext.setTaskContext(taskContext)
+
+            // Load the db and signal that we have entered state 1
+            stateLock.synchronized {
+              db.load(1)
+              state = 1
+              stateLock.notifyAll()
+            }
+
+            stateLock.synchronized {
+              // Wait until we have entered state 3, commit and signal
+              // that we have entered state 4
+              while (state != 3) {
+                stateLock.wait()
+              }
+              db.commit()
+              state = 4
+              stateLock.notifyAll()
+            }
+          }
+
+          stateLock.synchronized {
+            // Wait until we have entered state 1

Review Comment:
   Added additional comments to the test,  hope its clearer now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #48697: [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener
URL: https://github.com/apache/spark/pull/48697


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SS] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1825298252


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && acquiredThreadInfo != releaseForThreadOpt.get) {

Review Comment:
   It probably doesn't actually make a difference (the situation I described probably wouldn't happen because the completion listener runs on the task thread) but it is marginally better to use the entire object just in case. I will revert to using the entire object and add a note about this. Thanks for pointing it out. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1826334501


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.
+    //
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |

Review Comment:
   Done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823197978


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -743,17 +743,21 @@ class RocksDB(
    */
   def rollback(): Unit = {
     acquire(RollbackStore)
-    numKeysOnWritingVersion = numKeysOnLoadedVersion
-    loadedVersion = -1L
-    lastCommitBasedStateStoreCkptId = None
-    lastCommittedStateStoreCkptId = None
-    loadedStateStoreCkptId = None
-    sessionStateStoreCkptId = None
-    changelogWriter.foreach(_.abort())
-    // Make sure changelogWriter gets recreated next time.
-    changelogWriter = None
-    release(RollbackStore)
-    logInfo(log"Rolled back to ${MDC(LogKeys.VERSION_NUM, loadedVersion)}")
+    try {
+      numKeysOnWritingVersion = numKeysOnLoadedVersion
+      loadedVersion = -1L
+      lastCommitBasedStateStoreCkptId = None
+      lastCommittedStateStoreCkptId = None
+      loadedStateStoreCkptId = None
+      sessionStateStoreCkptId = None
+      changelogWriter.foreach(_.abort())
+      // Make sure changelogWriter gets recreated next time.
+      changelogWriter = None
+

Review Comment:
   i mean the newline - should we just remove it ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823188669


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -743,17 +743,21 @@ class RocksDB(
    */
   def rollback(): Unit = {
     acquire(RollbackStore)
-    numKeysOnWritingVersion = numKeysOnLoadedVersion
-    loadedVersion = -1L
-    lastCommitBasedStateStoreCkptId = None
-    lastCommittedStateStoreCkptId = None
-    loadedStateStoreCkptId = None
-    sessionStateStoreCkptId = None
-    changelogWriter.foreach(_.abort())
-    // Make sure changelogWriter gets recreated next time.
-    changelogWriter = None
-    release(RollbackStore)
-    logInfo(log"Rolled back to ${MDC(LogKeys.VERSION_NUM, loadedVersion)}")
+    try {
+      numKeysOnWritingVersion = numKeysOnLoadedVersion
+      loadedVersion = -1L
+      lastCommitBasedStateStoreCkptId = None
+      lastCommittedStateStoreCkptId = None
+      loadedStateStoreCkptId = None
+      sessionStateStoreCkptId = None
+      changelogWriter.foreach(_.abort())
+      // Make sure changelogWriter gets recreated next time.
+      changelogWriter = None
+

Review Comment:
   Yes so that if the task gets interrupted we still release. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on PR #48697:
URL: https://github.com/apache/spark/pull/48697#issuecomment-2450477488

   @liviazhu-db - i am not sure we need the latest commit. should we roll back to the prev version ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1824900931


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -787,9 +790,9 @@ class RocksDB(
 
   /** Release all resources */
   def close(): Unit = {
+    // Acquire DB instance lock and release at the end to allow for synchronized access
+    acquire(CloseStore)

Review Comment:
   Good point - this is how the DBR version is structured right now (https://src.dev.databricks.com/databricks-eng/runtime@12fc01e514f7b03e3732a110bcf9e4fdf7e390e7/-/blob/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala?L1125) but this is probably not ideal. I can wrap it in another try block that swallows the exception.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1824902849


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -903,8 +906,8 @@ class RocksDB(
    */
   def metricsOpt: Option[RocksDBMetrics] = {
     var rocksDBMetricsOpt: Option[RocksDBMetrics] = None
+    acquire(ReportStoreMetrics)
     try {

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1821920511


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -956,13 +956,43 @@ class RocksDB(
       acquiredThreadInfo = newAcquiredThreadInfo
       // Add a listener to always release the lock when the task (if active) completes
       Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
-        _ => this.release(StoreTaskCompletionListener)
+        _ => this.releaseForThread(StoreTaskCompletionListener, newAcquiredThreadInfo.threadRef)

Review Comment:
   this itself might not be enough right ?
   
   what if you have the following
   
   - load (lock)
   - release
   - close (lock)
   - completion listener release
   - close (release)
   
   in which case the last close is still resetting incorrectly ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1826184687


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -1001,6 +1031,10 @@ class RocksDB(
     }
   }
 
+  private[state] def getAcquiredThreadInfo(): AcquiredThreadInfo = acquireLock.synchronized {
+    acquiredThreadInfo.copy()

Review Comment:
   Done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823169045


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -956,51 +960,43 @@ class RocksDB(
       acquiredThreadInfo = newAcquiredThreadInfo
       // Add a listener to always release the lock when the task (if active) completes
       Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
-        _ => this.releaseForThread(StoreTaskCompletionListener, newAcquiredThreadInfo.threadRef)
+        _ => this.release(StoreTaskCompletionListener, Some(newAcquiredThreadInfo))
       })
       logInfo(log"RocksDB instance was acquired by ${MDC(LogKeys.THREAD, acquiredThreadInfo)} " +
         log"for opType=${MDC(LogKeys.OP_TYPE, opType.toString)}")
     }
   }
 
   /**
-   * Function to release RocksDB instance lock for if the provided thread is the one
-   * that acquired it.
+   * Function to release RocksDB instance lock that allows for synchronized access to the state
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
-   * @param threadRef - thread reference to check against the acquired thread
+   * @param forThreadOpt - optional thread to check against acquired thread
    */
-  private def releaseForThread(opType: RocksDBOpType,
-    threadRef: WeakReference[Thread]): Unit = acquireLock.synchronized {
-    if (acquiredThreadInfo != null && acquiredThreadInfo.threadRef.get.isDefined) {
-      if (!threadRef.get.isDefined) {
-        logInfo(log"Thread reference is empty when attempting to release for" +
-          log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
-          log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
-        return
-      }
+  private def release(opType: RocksDBOpType,
+    forThreadOpt: Option[AcquiredThreadInfo]): Unit = acquireLock.synchronized {

Review Comment:
   we can just use a default arg here with value as `None`. that way u don't need to pass it from all the callers ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823170885


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -956,51 +960,43 @@ class RocksDB(
       acquiredThreadInfo = newAcquiredThreadInfo
       // Add a listener to always release the lock when the task (if active) completes
       Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
-        _ => this.releaseForThread(StoreTaskCompletionListener, newAcquiredThreadInfo.threadRef)
+        _ => this.release(StoreTaskCompletionListener, Some(newAcquiredThreadInfo))
       })
       logInfo(log"RocksDB instance was acquired by ${MDC(LogKeys.THREAD, acquiredThreadInfo)} " +
         log"for opType=${MDC(LogKeys.OP_TYPE, opType.toString)}")
     }
   }
 
   /**
-   * Function to release RocksDB instance lock for if the provided thread is the one
-   * that acquired it.
+   * Function to release RocksDB instance lock that allows for synchronized access to the state
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
-   * @param threadRef - thread reference to check against the acquired thread
+   * @param forThreadOpt - optional thread to check against acquired thread
    */
-  private def releaseForThread(opType: RocksDBOpType,
-    threadRef: WeakReference[Thread]): Unit = acquireLock.synchronized {
-    if (acquiredThreadInfo != null && acquiredThreadInfo.threadRef.get.isDefined) {
-      if (!threadRef.get.isDefined) {
-        logInfo(log"Thread reference is empty when attempting to release for" +
-          log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
-          log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
-        return
-      }
+  private def release(opType: RocksDBOpType,
+    forThreadOpt: Option[AcquiredThreadInfo]): Unit = acquireLock.synchronized {

Review Comment:
   could we rename the arg as well to make it more descriptive ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823169615


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -956,51 +960,43 @@ class RocksDB(
       acquiredThreadInfo = newAcquiredThreadInfo
       // Add a listener to always release the lock when the task (if active) completes
       Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
-        _ => this.releaseForThread(StoreTaskCompletionListener, newAcquiredThreadInfo.threadRef)
+        _ => this.release(StoreTaskCompletionListener, Some(newAcquiredThreadInfo))
       })
       logInfo(log"RocksDB instance was acquired by ${MDC(LogKeys.THREAD, acquiredThreadInfo)} " +
         log"for opType=${MDC(LogKeys.OP_TYPE, opType.toString)}")
     }
   }
 
   /**
-   * Function to release RocksDB instance lock for if the provided thread is the one
-   * that acquired it.
+   * Function to release RocksDB instance lock that allows for synchronized access to the state
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
-   * @param threadRef - thread reference to check against the acquired thread
+   * @param forThreadOpt - optional thread to check against acquired thread
    */
-  private def releaseForThread(opType: RocksDBOpType,
-    threadRef: WeakReference[Thread]): Unit = acquireLock.synchronized {
-    if (acquiredThreadInfo != null && acquiredThreadInfo.threadRef.get.isDefined) {
-      if (!threadRef.get.isDefined) {
-        logInfo(log"Thread reference is empty when attempting to release for" +
-          log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
-          log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
-        return
-      }
+  private def release(opType: RocksDBOpType,

Review Comment:
   nit: lets move each arg to a new line ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823186798


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +969,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return

Review Comment:
   could we just do `if else` here ? i guess `return` is not recommended for scala per style guide ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1826336442


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.
+    //
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |

Review Comment:
   Added a new state so that I can un-nest threads 1 and 2 and then deleted thread 3. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1824931986


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -787,9 +790,9 @@ class RocksDB(
 
   /** Release all resources */
   def close(): Unit = {
+    // Acquire DB instance lock and release at the end to allow for synchronized access
+    acquire(CloseStore)

Review Comment:
   Reverted to the original version.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1824911528


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && acquiredThreadInfo != releaseForThreadOpt.get) {

Review Comment:
   I think that using the entire `ThreadInfo` is necessary here because we also want to compare the `TaskContext` and ensure that we are releasing it for the correct task. If we don't check this, I can imagine that we could run into some scenario where a new task is scheduled on the same thread and but the completion listener somehow runs later, releasing the new task's lock because it's on the same thread. I'm not _totally_ sure if this is possible.
   
   However, for `acquire` to be truly reentrant, we only want to compare thread IDs, maybe as a safeguard in case the `TaskContext` has changed somehow? I'm worried that we will cause a regression by changing this. Would love to hear @anishshri-db's thoughts. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1824911957


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && acquiredThreadInfo != releaseForThreadOpt.get) {

Review Comment:
   I think we should move away from this model anyway for the long term. but sure, for now - we can keep the comparisons consistent across acquire and release



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1826335367


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.
+    //
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |
+    // 1.    |                  | task complete    | wait for s3      |
+    //       |                  | signal s2, END   |                  |
+    // 2.    | start thread 3   |                  |                  | load, commit
+    //       | wait for timeout |                  |                  | timeout, END
+    //       | signal s3        |                  |                  |
+    // 3.    | wait for s4      |                  | commit           |
+    //       |                  |                  | signal s4, END   |
+    // 4.    | close db, END    |                  |                  |
+    //
+    // NOTE: state 3 and 4 are only for cleanup
+
+    // Create a custom ExecutionContext with 3 threads
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonFixedThreadPool(3, "pool-thread-executor"))
+    val timeout = 5.seconds
+
+    val stateLock = new Object()
+    var state = 0
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      val conf = dbConf.copy(lockAcquireTimeoutMs = timeout.toMillis.toInt)
+      val db = new RocksDB(
+        remoteDir,
+        conf = conf,
+        localRootDir = Utils.createTempDir(),
+        hadoopConf = new Configuration(),
+        loggingId = s"[Thread-${Thread.currentThread.getId}]",
+        useColumnFamilies = false
+      )
+      try {
+        Future { // THREAD 1
+          // Set thread 1's task context so that it is not a clone
+          // of the main thread's taskContext, which will end if the
+          // task is marked as complete
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          // Simulate a task that loads and commits, db should be unlocked after
+          db.load(0)
+          db.put("a", "1")
+          db.commit()
+
+          Future { // THREAD 2
+            // Set thread 2's task context so that it is not a clone of thread 1's
+            // so it won't be marked as complete
+            val taskContext = TaskContext.empty()
+            TaskContext.setTaskContext(taskContext)
+
+            // Load the db and signal that we have entered state 1
+            stateLock.synchronized {
+              db.load(1)
+
+              // Ensure we have the lock
+              assertAcquiredThreadIsCurrentThread(db)
+
+              state = 1
+              stateLock.notifyAll()
+            }
+
+            stateLock.synchronized {
+              // Wait until we have entered state 3 (thread 1 completed and
+              // thread 3 timed out waiting for our lock)
+              while (state != 3) {
+                stateLock.wait()
+              }
+
+              // Ensure we still have the lock
+              assertAcquiredThreadIsCurrentThread(db)
+
+              // commit and signal that we have entered state 4
+              db.commit()
+              state = 4
+              stateLock.notifyAll()
+            }
+          }
+
+          stateLock.synchronized {
+            // Wait until we have entered state 1 (thread 2 has loaded db and acquired lock)
+            while (state != 1) {
+              stateLock.wait()
+            }
+
+            // thread 1's task context is marked as complete and signal
+            // that we have entered state 2
+            // At this point, thread 2 should still hold the DB lock.
+            taskContext.markTaskCompleted(None)
+            state = 2
+            stateLock.notifyAll()
+          }
+        }
+
+        stateLock.synchronized {
+          // Wait until we have entered state 2 (thread 1 is complete)
+          while (state != 2) {
+            stateLock.wait()
+          }
+        }
+
+        // Thread 3 tries to load db, but it is still locked by thread 2
+        val thread3 = Future {
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          db.load(1)
+          db.commit()
+        }
+
+        // Therefore it should timeout
+        val e = intercept[SparkException] {
+          ThreadUtils.awaitResult(thread3, 10.seconds)
+        }
+        assert(e.getCause.getMessage.contains(
+          "CANNOT_LOAD_STATE_STORE.UNRELEASED_THREAD_ERROR"))
+
+        stateLock.synchronized {
+          // Signal that we have entered state 3 (thread 2 can now release lock)
+          state = 3
+          stateLock.notifyAll()
+        }
+
+        stateLock.synchronized {
+          // Wait until we have entered state 4 (thread 2 has released lock)
+          // so that we can clean up
+          while (state != 4) {
+            stateLock.wait()
+          }
+        }
+      } finally {
+        db.close()
+      }
+    }
+  }
+
+  test("RocksDB task completion listener correctly releases for failed task") {
+    // This test verifies that a thread that locks the DB and then fails
+    // can rely on the completion listener to release the lock.
+
+    // Create a custom ExecutionContext with 1 thread
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonSingleThreadExecutor("single-thread-executor"))
+    val timeout = 5.seconds
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      withDB(remoteDir) { db =>
+        // Release the lock acquired by withDB
+        db.commit()
+
+        // New task that will load and then complete with failure
+        val fut = Future {
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          db.load(0)
+          assertAcquiredThreadIsCurrentThread(db)
+
+          // Task completion listener should unlock
+          taskContext.markTaskCompleted(
+            Some(new SparkException("Task failure injection")))
+        }
+
+        ThreadUtils.awaitResult(fut, timeout)
+
+        // We should be able to load the db here because the

Review Comment:
   Checked lock using current lock information instead of waiting for timeout. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823178303


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -956,51 +960,43 @@ class RocksDB(
       acquiredThreadInfo = newAcquiredThreadInfo
       // Add a listener to always release the lock when the task (if active) completes
       Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
-        _ => this.releaseForThread(StoreTaskCompletionListener, newAcquiredThreadInfo.threadRef)
+        _ => this.release(StoreTaskCompletionListener, Some(newAcquiredThreadInfo))
       })
       logInfo(log"RocksDB instance was acquired by ${MDC(LogKeys.THREAD, acquiredThreadInfo)} " +
         log"for opType=${MDC(LogKeys.OP_TYPE, opType.toString)}")
     }
   }
 
   /**
-   * Function to release RocksDB instance lock for if the provided thread is the one
-   * that acquired it.
+   * Function to release RocksDB instance lock that allows for synchronized access to the state
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
-   * @param threadRef - thread reference to check against the acquired thread
+   * @param forThreadOpt - optional thread to check against acquired thread
    */
-  private def releaseForThread(opType: RocksDBOpType,
-    threadRef: WeakReference[Thread]): Unit = acquireLock.synchronized {
-    if (acquiredThreadInfo != null && acquiredThreadInfo.threadRef.get.isDefined) {
-      if (!threadRef.get.isDefined) {
-        logInfo(log"Thread reference is empty when attempting to release for" +
-          log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
-          log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
-        return
-      }
+  private def release(opType: RocksDBOpType,
+    forThreadOpt: Option[AcquiredThreadInfo]): Unit = acquireLock.synchronized {

Review Comment:
   Is `releaseForThreadOpt` sufficiently descriptive? `currAcquiredThreadInfoOpt` could be mixed up with `acquiredThreadInfo` too easily.



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +969,34 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param forThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(opType: RocksDBOpType,
+    forThreadOpt: Option[AcquiredThreadInfo]): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (forThreadOpt.nonEmpty) {
+        if (forThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && forThreadOpt.get.threadRef.get.isDefined
+          && acquiredThreadInfo.threadRef.get.get.getId

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823176277


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -956,51 +960,43 @@ class RocksDB(
       acquiredThreadInfo = newAcquiredThreadInfo
       // Add a listener to always release the lock when the task (if active) completes
       Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
-        _ => this.releaseForThread(StoreTaskCompletionListener, newAcquiredThreadInfo.threadRef)
+        _ => this.release(StoreTaskCompletionListener, Some(newAcquiredThreadInfo))
       })
       logInfo(log"RocksDB instance was acquired by ${MDC(LogKeys.THREAD, acquiredThreadInfo)} " +
         log"for opType=${MDC(LogKeys.OP_TYPE, opType.toString)}")
     }
   }
 
   /**
-   * Function to release RocksDB instance lock for if the provided thread is the one
-   * that acquired it.
+   * Function to release RocksDB instance lock that allows for synchronized access to the state
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
-   * @param threadRef - thread reference to check against the acquired thread
+   * @param forThreadOpt - optional thread to check against acquired thread
    */
-  private def releaseForThread(opType: RocksDBOpType,
-    threadRef: WeakReference[Thread]): Unit = acquireLock.synchronized {
-    if (acquiredThreadInfo != null && acquiredThreadInfo.threadRef.get.isDefined) {
-      if (!threadRef.get.isDefined) {
-        logInfo(log"Thread reference is empty when attempting to release for" +
-          log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
-          log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
-        return
-      }
+  private def release(opType: RocksDBOpType,
+    forThreadOpt: Option[AcquiredThreadInfo]): Unit = acquireLock.synchronized {

Review Comment:
   Right, forgot we could do that, done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1825410674


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -1001,6 +1031,10 @@ class RocksDB(
     }
   }
 
+  private[state] def getAcquiredThreadInfo(): AcquiredThreadInfo = acquireLock.synchronized {
+    acquiredThreadInfo.copy()

Review Comment:
   acquiredThreadInfo could be null - you'd want to return Option instead in general.



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,39 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {

Review Comment:
   Please define the variable as it's long enough and referred multiple times.
   
   ```
   val releaseForThread = releaseForThreadOpt.get
   ```
   
   We could even give a try with pattern matching (it's better with your change for AcquiredThreadInfo as all values are placed in constructor params) but it's also fine for me to leave it as it is.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.
+    //
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |
+    // 1.    |                  | task complete    | wait for s3      |
+    //       |                  | signal s2, END   |                  |
+    // 2.    | start thread 3   |                  |                  | load, commit
+    //       | wait for timeout |                  |                  | timeout, END
+    //       | signal s3        |                  |                  |
+    // 3.    | wait for s4      |                  | commit           |
+    //       |                  |                  | signal s4, END   |
+    // 4.    | close db, END    |                  |                  |
+    //
+    // NOTE: state 3 and 4 are only for cleanup
+
+    // Create a custom ExecutionContext with 3 threads
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonFixedThreadPool(3, "pool-thread-executor"))
+    val timeout = 5.seconds
+
+    val stateLock = new Object()
+    var state = 0
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      val conf = dbConf.copy(lockAcquireTimeoutMs = timeout.toMillis.toInt)
+      val db = new RocksDB(
+        remoteDir,
+        conf = conf,
+        localRootDir = Utils.createTempDir(),
+        hadoopConf = new Configuration(),
+        loggingId = s"[Thread-${Thread.currentThread.getId}]",
+        useColumnFamilies = false
+      )
+      try {
+        Future { // THREAD 1
+          // Set thread 1's task context so that it is not a clone
+          // of the main thread's taskContext, which will end if the
+          // task is marked as complete
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          // Simulate a task that loads and commits, db should be unlocked after
+          db.load(0)
+          db.put("a", "1")
+          db.commit()
+
+          Future { // THREAD 2
+            // Set thread 2's task context so that it is not a clone of thread 1's
+            // so it won't be marked as complete
+            val taskContext = TaskContext.empty()
+            TaskContext.setTaskContext(taskContext)
+
+            // Load the db and signal that we have entered state 1
+            stateLock.synchronized {
+              db.load(1)
+
+              // Ensure we have the lock
+              assertAcquiredThreadIsCurrentThread(db)
+
+              state = 1
+              stateLock.notifyAll()
+            }
+
+            stateLock.synchronized {
+              // Wait until we have entered state 3 (thread 1 completed and
+              // thread 3 timed out waiting for our lock)
+              while (state != 3) {
+                stateLock.wait()
+              }
+
+              // Ensure we still have the lock
+              assertAcquiredThreadIsCurrentThread(db)
+
+              // commit and signal that we have entered state 4
+              db.commit()
+              state = 4
+              stateLock.notifyAll()
+            }
+          }
+
+          stateLock.synchronized {

Review Comment:
   It's not very intuitive for reader whether below lines are bound to thread 1 or 2, or even main thread. Let's say, you are reading the code sequentially - you see the code about thread 1 what to do in state 0, and jump to thread 2 about what to do in state 0, and after that, we are dealing with state 3 with thread 2. Once the definition of thread 2 completes, we go back to state 0 in thread 1. 
   
   time and threads are mixed up; I understand this is probably some effort trying to describe the behavior based on time via interleaving threads, but still there is a back and forth in time. Since we have a table explaining which thread will do which behavior for which state, I think just making the definition of all threads be sequential without interleaving would be less confusing.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.
+    //
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |
+    // 1.    |                  | task complete    | wait for s3      |
+    //       |                  | signal s2, END   |                  |
+    // 2.    | start thread 3   |                  |                  | load, commit
+    //       | wait for timeout |                  |                  | timeout, END
+    //       | signal s3        |                  |                  |
+    // 3.    | wait for s4      |                  | commit           |
+    //       |                  |                  | signal s4, END   |
+    // 4.    | close db, END    |                  |                  |
+    //
+    // NOTE: state 3 and 4 are only for cleanup
+
+    // Create a custom ExecutionContext with 3 threads
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonFixedThreadPool(3, "pool-thread-executor"))
+    val timeout = 5.seconds
+
+    val stateLock = new Object()
+    var state = 0
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      val conf = dbConf.copy(lockAcquireTimeoutMs = timeout.toMillis.toInt)
+      val db = new RocksDB(
+        remoteDir,
+        conf = conf,
+        localRootDir = Utils.createTempDir(),
+        hadoopConf = new Configuration(),
+        loggingId = s"[Thread-${Thread.currentThread.getId}]",
+        useColumnFamilies = false
+      )
+      try {
+        Future { // THREAD 1
+          // Set thread 1's task context so that it is not a clone
+          // of the main thread's taskContext, which will end if the
+          // task is marked as complete
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          // Simulate a task that loads and commits, db should be unlocked after
+          db.load(0)
+          db.put("a", "1")
+          db.commit()
+
+          Future { // THREAD 2
+            // Set thread 2's task context so that it is not a clone of thread 1's
+            // so it won't be marked as complete
+            val taskContext = TaskContext.empty()
+            TaskContext.setTaskContext(taskContext)
+
+            // Load the db and signal that we have entered state 1
+            stateLock.synchronized {
+              db.load(1)
+
+              // Ensure we have the lock
+              assertAcquiredThreadIsCurrentThread(db)
+
+              state = 1
+              stateLock.notifyAll()
+            }
+
+            stateLock.synchronized {
+              // Wait until we have entered state 3 (thread 1 completed and
+              // thread 3 timed out waiting for our lock)
+              while (state != 3) {
+                stateLock.wait()
+              }
+
+              // Ensure we still have the lock
+              assertAcquiredThreadIsCurrentThread(db)
+
+              // commit and signal that we have entered state 4
+              db.commit()
+              state = 4
+              stateLock.notifyAll()
+            }
+          }
+
+          stateLock.synchronized {
+            // Wait until we have entered state 1 (thread 2 has loaded db and acquired lock)
+            while (state != 1) {
+              stateLock.wait()
+            }
+
+            // thread 1's task context is marked as complete and signal
+            // that we have entered state 2
+            // At this point, thread 2 should still hold the DB lock.
+            taskContext.markTaskCompleted(None)
+            state = 2
+            stateLock.notifyAll()
+          }
+        }
+
+        stateLock.synchronized {
+          // Wait until we have entered state 2 (thread 1 is complete)
+          while (state != 2) {
+            stateLock.wait()
+          }
+        }
+
+        // Thread 3 tries to load db, but it is still locked by thread 2
+        val thread3 = Future {
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          db.load(1)
+          db.commit()
+        }
+
+        // Therefore it should timeout
+        val e = intercept[SparkException] {
+          ThreadUtils.awaitResult(thread3, 10.seconds)
+        }
+        assert(e.getCause.getMessage.contains(
+          "CANNOT_LOAD_STATE_STORE.UNRELEASED_THREAD_ERROR"))
+
+        stateLock.synchronized {
+          // Signal that we have entered state 3 (thread 2 can now release lock)
+          state = 3
+          stateLock.notifyAll()
+        }
+
+        stateLock.synchronized {
+          // Wait until we have entered state 4 (thread 2 has released lock)
+          // so that we can clean up
+          while (state != 4) {
+            stateLock.wait()
+          }
+        }
+      } finally {
+        db.close()
+      }
+    }
+  }
+
+  test("RocksDB task completion listener correctly releases for failed task") {
+    // This test verifies that a thread that locks the DB and then fails
+    // can rely on the completion listener to release the lock.
+
+    // Create a custom ExecutionContext with 1 thread
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonSingleThreadExecutor("single-thread-executor"))
+    val timeout = 5.seconds
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      withDB(remoteDir) { db =>
+        // Release the lock acquired by withDB
+        db.commit()
+
+        // New task that will load and then complete with failure
+        val fut = Future {
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          db.load(0)
+          assertAcquiredThreadIsCurrentThread(db)
+
+          // Task completion listener should unlock
+          taskContext.markTaskCompleted(
+            Some(new SparkException("Task failure injection")))
+        }
+
+        ThreadUtils.awaitResult(fut, timeout)
+
+        // We should be able to load the db here because the

Review Comment:
   nit: Is it intended to use load() for verification? Since we now have a way to retrieve the current lock information. The only viable scenario we need to use load() is something like we still need to wait more here so that other thread may unlock the RocksDB instance. Is it the case?
   
   This is because we had to work on reducing the time of tests, and we may not want to work this globally again.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.

Review Comment:
   If we are able to see the lock status from RocksDB (it's possible with this PR), why not just check in state 2 in the main thread rather than spinning up thread 3? I think same applies to the next test.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.
+    //
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |

Review Comment:
   nit: maybe easier to look if we have explicit separation among states. bar, or empty line.



##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.
+    //
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |

Review Comment:
   Also, why not just splitting more state to not rely on the order of execution among threads in the same state? I meant, each state handles an execution of a single thread.
   
   For example, I see state 0 is implemented with nested thread submission to ensure the execution order between thread 1 and thread 2. Personally this is harder to read, because you have implemented thread 2 within the implementation of thread 1, and thread 2 is expected to be ended "after" thread 1 has ended. The code scope is different from execution scope and I think it's just more intuitive to let all threads to be completely parallel but just dependent to the state.



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +969,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return

Review Comment:
   You may want to read the PR description of this PR.
   https://github.com/apache/spark/pull/48309
   
   I don't know whether this return would ever be possible to be non-local, but if we have alternative approach to make the code similarly clean without return, we'd like to take the alternative.
   
   For example, additional var could work to track whether the lock should be released or not.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1826336625


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,200 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // This test verifies that a thread that locks then unlocks the db and then
+    // fires a completion listener (Thread 1) does not unlock the lock validly
+    // acquired by another thread (Thread 2).
+    // Thread 3 verifies that Thread 2 still holds the lock by timing out.
+    //
+    // Timeline of this test:
+    // STATE | MAIN             | THREAD 1         | THREAD 2         | THREAD 3
+    // ------| ---------------- | ---------------- | ---------------- | ----------------
+    // 0.    | wait for s2      | Load, commit     |                  |
+    //       |                  | wait for s1      | load, signal s1  |
+    // 1.    |                  | task complete    | wait for s3      |
+    //       |                  | signal s2, END   |                  |
+    // 2.    | start thread 3   |                  |                  | load, commit
+    //       | wait for timeout |                  |                  | timeout, END
+    //       | signal s3        |                  |                  |
+    // 3.    | wait for s4      |                  | commit           |
+    //       |                  |                  | signal s4, END   |
+    // 4.    | close db, END    |                  |                  |
+    //
+    // NOTE: state 3 and 4 are only for cleanup
+
+    // Create a custom ExecutionContext with 3 threads
+    implicit val ec: ExecutionContext = ExecutionContext.fromExecutor(
+      ThreadUtils.newDaemonFixedThreadPool(3, "pool-thread-executor"))
+    val timeout = 5.seconds
+
+    val stateLock = new Object()
+    var state = 0
+
+    withTempDir { dir =>
+      val remoteDir = dir.getCanonicalPath
+      val conf = dbConf.copy(lockAcquireTimeoutMs = timeout.toMillis.toInt)
+      val db = new RocksDB(
+        remoteDir,
+        conf = conf,
+        localRootDir = Utils.createTempDir(),
+        hadoopConf = new Configuration(),
+        loggingId = s"[Thread-${Thread.currentThread.getId}]",
+        useColumnFamilies = false
+      )
+      try {
+        Future { // THREAD 1
+          // Set thread 1's task context so that it is not a clone
+          // of the main thread's taskContext, which will end if the
+          // task is marked as complete
+          val taskContext = TaskContext.empty()
+          TaskContext.setTaskContext(taskContext)
+
+          // Simulate a task that loads and commits, db should be unlocked after
+          db.load(0)
+          db.put("a", "1")
+          db.commit()
+
+          Future { // THREAD 2
+            // Set thread 2's task context so that it is not a clone of thread 1's
+            // so it won't be marked as complete
+            val taskContext = TaskContext.empty()
+            TaskContext.setTaskContext(taskContext)
+
+            // Load the db and signal that we have entered state 1
+            stateLock.synchronized {
+              db.load(1)
+
+              // Ensure we have the lock
+              assertAcquiredThreadIsCurrentThread(db)
+
+              state = 1
+              stateLock.notifyAll()
+            }
+
+            stateLock.synchronized {
+              // Wait until we have entered state 3 (thread 1 completed and
+              // thread 3 timed out waiting for our lock)
+              while (state != 3) {
+                stateLock.wait()
+              }
+
+              // Ensure we still have the lock
+              assertAcquiredThreadIsCurrentThread(db)
+
+              // commit and signal that we have entered state 4
+              db.commit()
+              state = 4
+              stateLock.notifyAll()
+            }
+          }
+
+          stateLock.synchronized {

Review Comment:
   Same as above comment and added some more comments that hopefully make execution order clearer!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823172962


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +969,34 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param forThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(opType: RocksDBOpType,
+    forThreadOpt: Option[AcquiredThreadInfo]): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (forThreadOpt.nonEmpty) {
+        if (forThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && forThreadOpt.get.threadRef.get.isDefined
+          && acquiredThreadInfo.threadRef.get.get.getId

Review Comment:
   can we not do object level equality comparison here ? or maybe add a comparator function if required ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823003624


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -956,13 +956,43 @@ class RocksDB(
       acquiredThreadInfo = newAcquiredThreadInfo
       // Add a listener to always release the lock when the task (if active) completes
       Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit] {
-        _ => this.release(StoreTaskCompletionListener)
+        _ => this.releaseForThread(StoreTaskCompletionListener, newAcquiredThreadInfo.threadRef)

Review Comment:
   For the short-term fix, I think we have to assume that the completion listener will only trigger at the end of a task or else the whole thing doesn't work. If we assume that it could be triggered at any time, then even the following situation would break: 
   * load(lock)
   * completion listener release
   * now we are unsafe



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1824909289


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -787,9 +790,9 @@ class RocksDB(
 
   /** Release all resources */
   def close(): Unit = {
+    // Acquire DB instance lock and release at the end to allow for synchronized access
+    acquire(CloseStore)

Review Comment:
   yes - if we cannot acquire the lock - we should fail the task



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -903,8 +906,8 @@ class RocksDB(
    */
   def metricsOpt: Option[RocksDBMetrics] = {
     var rocksDBMetricsOpt: Option[RocksDBMetrics] = None
+    acquire(ReportStoreMetrics)
     try {

Review Comment:
   yes same here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823187181


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -743,17 +743,21 @@ class RocksDB(
    */
   def rollback(): Unit = {
     acquire(RollbackStore)
-    numKeysOnWritingVersion = numKeysOnLoadedVersion
-    loadedVersion = -1L
-    lastCommitBasedStateStoreCkptId = None
-    lastCommittedStateStoreCkptId = None
-    loadedStateStoreCkptId = None
-    sessionStateStoreCkptId = None
-    changelogWriter.foreach(_.abort())
-    // Make sure changelogWriter gets recreated next time.
-    changelogWriter = None
-    release(RollbackStore)
-    logInfo(log"Rolled back to ${MDC(LogKeys.VERSION_NUM, loadedVersion)}")
+    try {
+      numKeysOnWritingVersion = numKeysOnLoadedVersion
+      loadedVersion = -1L
+      lastCommitBasedStateStoreCkptId = None
+      lastCommittedStateStoreCkptId = None
+      loadedStateStoreCkptId = None
+      sessionStateStoreCkptId = None
+      changelogWriter.foreach(_.abort())
+      // Make sure changelogWriter gets recreated next time.
+      changelogWriter = None
+

Review Comment:
   intentional ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "siying (via GitHub)" <gi...@apache.org>.
siying commented on PR #48697:
URL: https://github.com/apache/spark/pull/48697#issuecomment-2448660759

   Instead of threadRef, can we generate a uniqueID instead for a specific acquire/release pair. ThreadRef technically isn't stable. I don't think threadRef will cause any bug but I just feel like a solution depending on which thread can release the lock the thread acquired isn't robust enough.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #48697:
URL: https://github.com/apache/spark/pull/48697#issuecomment-2453214807

   Looks like CI only failed with docker integration which is unrelated.
   https://github.com/liviazhu-db/spark/runs/32409391731


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163][SS] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #48697:
URL: https://github.com/apache/spark/pull/48697#issuecomment-2453214829

   Thanks! Merging to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1823194669


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +969,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return

Review Comment:
   Are you referring to https://github.com/databricks/scala-style-guide?tab=readme-ov-file#return-statements? I think this falls under "Use `return` as a guard to simplify control flow without adding a level of indentation"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SS] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1825281283


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,166 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // Create a custom ExecutionContext with 3 threads

Review Comment:
   hmm, still showing 5 for me: `ThreadUtils.newDaemonFixedThreadPool(5, "pool-thread-executor"))`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SS] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1825295966


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -2240,6 +2240,166 @@ class RocksDBSuite extends AlsoTestWithChangelogCheckpointingEnabled with Shared
     }
   }
 
+  test("Rocks DB task completion listener does not double unlock acquireThread") {
+    // Create a custom ExecutionContext with 3 threads

Review Comment:
   Oh sorry, actually fixed now 😆 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SS] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #48697:
URL: https://github.com/apache/spark/pull/48697#discussion_r1825252564


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -965,12 +968,33 @@ class RocksDB(
 
   /**
    * Function to release RocksDB instance lock that allows for synchronized access to the state
-   * store instance
+   * store instance. Optionally provide a thread to check against, and release only if provided
+   * thread is the one that acquired the lock.
    *
    * @param opType - operation type releasing the lock
+   * @param releaseForThreadOpt - optional thread to check against acquired thread
    */
-  private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
+  private def release(
+      opType: RocksDBOpType,
+      releaseForThreadOpt: Option[AcquiredThreadInfo] = None): Unit = acquireLock.synchronized {
     if (acquiredThreadInfo != null) {
+      if (releaseForThreadOpt.nonEmpty) {
+        if (releaseForThreadOpt.get.threadRef.get.isEmpty) {
+          logInfo(log"Thread reference is empty when attempting to release for" +
+            log" opType=${MDC(LogKeys.OP_TYPE, opType.toString)}, ignoring release." +
+            log" Lock is held by ${MDC(LogKeys.THREAD, acquiredThreadInfo)}")
+          return
+        }
+
+        if (acquiredThreadInfo.threadRef.get.isDefined
+          && acquiredThreadInfo != releaseForThreadOpt.get) {

Review Comment:
   @liviazhu-db I agree that comparing the entire object is a better pattern, my question here was more about why they are different, just to make sure we know why and not missing something. Don't want us to be consistent for just consistency sake. If we believe comparing the full object is the safer approach then lets do that. We can just do that for only this release func for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-50163] Fix the RocksDB extra acquireLock release due to the completion listener [spark]

Posted by "liviazhu-db (via GitHub)" <gi...@apache.org>.
liviazhu-db commented on PR #48697:
URL: https://github.com/apache/spark/pull/48697#issuecomment-2448682549

   @siying, @anishshri-db and I discussed this but it will be difficult to propagate this uniqueID across `load` and `commit`/`commitAsync` calls (so that the `load` acquire pass the id to the `commit` release). This will likely need a higher-level API change which we can consider for a larger long-term fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org