[jira] [Updated] (SPARK-27666) Do not release lock while TaskContext already completed

["wuyi (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-27666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

wuyi updated SPARK-27666:
-------------------------
    Summary: Do not release lock while TaskContext already completed  (was: Stop python runner threads when task finishes)

> Do not release lock while TaskContext already completed
> -------------------------------------------------------
>
>                 Key: SPARK-27666
>                 URL: https://issues.apache.org/jira/browse/SPARK-27666
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Wenchen Fan
>            Priority: Major
>
> When a task finishes, we should stop the threads that are spawned by the Python runner immediately.
> https://github.com/apache/spark/pull/24542 fixed the problem that, if the Python runner thread releases the block lock at the end, and the block lock is already released when task finishes, we will hit assertion error.
> If we can stop the python runner thread when task finishes, we don't need https://github.com/apache/spark/pull/24542 any more.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org