[jira] [Created] (SPARK-54217) PythonRunner does not synchronize Python worker kill/release decisions

["Steven Tran (Jira)" <ji...@apache.org>]

Steven Tran created SPARK-54217:
-----------------------------------

             Summary: PythonRunner does not synchronize Python worker kill/release decisions
                 Key: SPARK-54217
                 URL: https://issues.apache.org/jira/browse/SPARK-54217
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 4.2.0
            Reporter: Steven Tran


`PythonWorkerFactory` in daemon mode will facilitate worker reuse, where possible, as long as the worker successfully completed its last-assigned task (via `releasePythonWorker`). The worker will be released into the idle queue to be picked up by the next `createPythonWorker` call.

 

However, there is a race condition that can result in a released worker in the`PythonWorkerFactory` idle queue getting killed. i.e. the `PythonRunner` lacks synchronization between:
 * the main task thread's decision to release its associated Python worker (when work is complete), and
 * the `MonitorThread`'s decision to kill the associated Python worker (when requested by the executor, e.g. speculative execution where another attempt succeeds).

So, the following sequence of events is possible:
 # `PythonRunner` is running
 # The Python worker finishes its work and writes `END_OF_STREAM` to signal back to `PythonRunner`'s main task thread that it is done
 # `PythonRunner`'s main task thread receives this instruction and releases the worker for reuse
 # Executor decides to kill this task (e.g. speculative execution)
 # `PythonRunner`'s `MonitorThread` receives this instruction and kills the already-relinquished `PythonWorker`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org