[jira] [Updated] (SPARK-60222) Keep task completion listeners on the completing thread when another thread invokes listeners

["L. C. Hsieh (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

L. C. Hsieh updated SPARK-60222:
--------------------------------
    Affects Version/s: 4.3.0
                       4.4.0
                           (was: 5.0.0)

> Keep task completion listeners on the completing thread when another thread invokes listeners
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-60222
>                 URL: https://issues.apache.org/jira/browse/SPARK-60222
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 4.3.0, 4.4.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>
> TaskContextImpl runs task listeners on one thread at a time, tracked by listenerInvocationThread. In two cases, the completion listeners do not run on the thread that completes the task:
> 1. When another thread is invoking listeners as the task completes, markTaskCompleted returns without running any completion listener. For example, Task.kill calls markInterrupted on the kill thread, and if an interrupt listener throws, markTaskFailed runs the failure listeners on that thread. The completion listeners are then left until some other thread adds a listener, or skipped entirely. The task can then reach cleanUpAllAllocatedMemory while resources that the completion listeners would release are still in use. The ErrorNotifier doc and a StreamingShuffleSuite test describe this race.
> 2. When a listener throws, invokeListeners clears listenerInvocationThread while markTaskFailed runs the failure listeners, and reclaims it afterwards. Another thread that adds a listener in between can take over running the pending listeners, possibly concurrently with this thread.
> Failure listeners that another thread queues while a thread runs the completion listeners, and completion listeners queued while a thread runs failure listeners after completion, are also dropped.
> This matters for helper threads that the completion listeners stop and wait for, such as the pipelined Python UDF writer (SPARK-60113): if the helper thread adds a listener lazily at such a moment, it ends up running the listener that waits for itself.
> The proposed fix keeps the completion listeners on the completing thread: markTaskCompleted waits for another thread invoking listeners and takes the invocation over directly, failure listeners run inside the current invocation when a listener throws, and a thread runs the listeners left to it before it releases the invocation. This was first done in apache/spark#59332 and split out of it during review.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org