[jira] [Created] (SPARK-39847) Race condition related to interruption of task threads while they are in RocksDBLoader.loadLibrary()

["Josh Rosen (Jira)" <ji...@apache.org>]

Josh Rosen created SPARK-39847:
----------------------------------

             Summary: Race condition related to interruption of task threads while they are in RocksDBLoader.loadLibrary()
                 Key: SPARK-39847
                 URL: https://issues.apache.org/jira/browse/SPARK-39847
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 3.2.0
            Reporter: Josh Rosen
            Assignee: Josh Rosen


One of our workloads experienced a rare failure in `RocksDBLoader`
{code:java}
Caused by: java.lang.IllegalThreadStateException
	at java.lang.Thread.start(Thread.java:708)
	at org.apache.spark.sql.execution.streaming.state.RocksDBLoader$.loadLibrary(RocksDBLoader.scala:51) {code}
After investigation, we determined that was due to task cancellation: if the task which starts the RocksDB library loading is interrupted, another thread may begin a load and crash with the thread state exception.

Skimming through the code in [RocksDBLoader|https://github.com/databricks/runtime/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBLoader.scala], I spot a potential race condition:
 * Although the native JNI call is uninterruptible, the thread which calls loadLibrary is still interruptible. Let’s call that thread the “task thread”.

 * Say we have two tasks, A and B, which both want to load the JNI library.

 * Say that Task A wins the race to perform the load and enters the synchronized block in loadLibrary(), spawns a child thread to perform the actual loading, then blocks in the loadLibraryThread.join() call.

 * If Task A is interrupted, an InterruptedException will be thrown and it will exit the loadLibrary synchronized block.

 * At this point, Task B enters the synchronized block and sees that exception == null because the loading thread is still running, so it calls loadLibraryThread.start() and hits the thread state error.

One way to fix this is to add
{code:java}
 if (loadLibraryThread.getState == Thread.State.NEW) {
        loadLibraryThread.start()
      }{code}
to ensure that only one thread starts the loadLibraryThread. If the original starter thread is interrupted then a new thread will encounter this block, skip the start(), proceed to the join() and block on the original load thread.

I will submit a PR with this fix.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org