[jira] [Commented] (SPARK-35498) Add an API "inheritable_thread_target" which return a wrapped thread target for pyspark pin thread mode

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-35498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350272#comment-17350272 ] 

Apache Spark commented on SPARK-35498:
--------------------------------------

User 'WeichenXu123' has created a pull request for this issue:
https://github.com/apache/spark/pull/32644

> Add an API "inheritable_thread_target" which return a wrapped thread target for pyspark pin thread mode
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-35498
>                 URL: https://issues.apache.org/jira/browse/SPARK-35498
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 3.2.0
>            Reporter: Weichen Xu
>            Priority: Major
>
> In pyspark, user may create some threads, not via `Thread` object, but via some parallel helper function such as:
> `thread_pool.imap_unordered`
> In this case, we need create a function wrapper, used in pin thread mode, The wrapper function, before calling original thread target, it inherits the inheritable properties specific to JVM thread such as ``InheritableThreadLocal``, and after original thread target return, garbage-collects the Python thread instance and also closes the connection which finishes JVM thread correctly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org