[jira] [Created] (SPARK-45942) Only do the thread interruption check for putIterator on executors

["Huanli Wang (Jira)" <ji...@apache.org>]

Huanli Wang created SPARK-45942:
-----------------------------------

             Summary: Only do the thread interruption check for putIterator on executors
                 Key: SPARK-45942
                 URL: https://issues.apache.org/jira/browse/SPARK-45942
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 4.0.0
            Reporter: Huanli Wang


https://issues.apache.org/jira/browse/SPARK-45025 

introduces a peaceful thread interruption handling. However, there is an edge case: when a streaming query is stopped on the driver, it interrupts the stream execution thread. If the streaming query is doing memory store operations on driver and performs {{doPutIterator}} at the same time, the [unroll process will be broken|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L224] and [returns used memory|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L245-L247].

This can result in {{closeChannelException}} as it falls into this [case clause|https://github.com/apache/spark/blob/aa646d3050028272f7333deaef52f20e6975e0ed/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1614-L1622] which opens an I/O channel and persists the data into the disk. However, because the thread is interrupted, the channel will be closed at the begin: [https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L172] and throws out {{closeChannelException}}

On executors, [the task will be killed if the thread is interrupted|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L374], however, we don't do it on the driver.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org