[jira] [Commented] (SPARK-39847) Race condition related to interruption of task threads while they are in RocksDBLoader.loadLibrary()

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-39847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17570343#comment-17570343 ] 

Apache Spark commented on SPARK-39847:
--------------------------------------

User 'JoshRosen' has created a pull request for this issue:
https://github.com/apache/spark/pull/37260

> Race condition related to interruption of task threads while they are in RocksDBLoader.loadLibrary()
> ----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-39847
>                 URL: https://issues.apache.org/jira/browse/SPARK-39847
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.2.0
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>            Priority: Major
>
> One of our workloads experienced a rare failure in `RocksDBLoader`
> {code:java}
> Caused by: java.lang.IllegalThreadStateException
> 	at java.lang.Thread.start(Thread.java:708)
> 	at org.apache.spark.sql.execution.streaming.state.RocksDBLoader$.loadLibrary(RocksDBLoader.scala:51) {code}
> After investigation, we determined that was due to task cancellation: if the task which starts the RocksDB library loading is interrupted, another thread may begin a load and crash with the thread state exception.
> Skimming through the code in [RocksDBLoader|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBLoader.scala], I spot a potential race condition:
>  * Although the native JNI call is uninterruptible, the thread which calls loadLibrary is still interruptible. Let’s call that thread the “task thread”.
>  * Say we have two tasks, A and B, which both want to load the JNI library.
>  * Say that Task A wins the race to perform the load and enters the synchronized block in loadLibrary(), spawns a child thread to perform the actual loading, then blocks in the loadLibraryThread.join() call.
>  * If Task A is interrupted, an InterruptedException will be thrown and it will exit the loadLibrary synchronized block.
>  * At this point, Task B enters the synchronized block and sees that exception == null because the loading thread is still running, so it calls loadLibraryThread.start() and hits the thread state error.
> One way to fix this is to add
> {code:java}
>  if (loadLibraryThread.getState == Thread.State.NEW) {
>         loadLibraryThread.start()
>       }{code}
> to ensure that only one thread starts the loadLibraryThread. If the original starter thread is interrupted then a new thread will encounter this block, skip the start(), proceed to the join() and block on the original load thread.
> I will submit a PR with this fix.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org