[jira] [Resolved] (SPARK-59233) Pass the session's Python worker environment to every Python function family

["Ruifeng Zheng (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ruifeng Zheng resolved SPARK-59233.
-----------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 58510
[https://github.com/apache/spark/pull/58510]

> Pass the session's Python worker environment to every Python function family
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-59233
>                 URL: https://issues.apache.org/jira/browse/SPARK-59233
>             Project: Spark
>          Issue Type: Task
>          Components: PySpark, SQL
>    Affects Versions: 5.0.0, 4.4.0
>            Reporter: Kaicheng Zhou
>            Assignee: Kaicheng Zhou
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> SPARK-58752 added the {{spark.pythonWorkerEnv.*}} session configurations, which set environment variables for the Python workers that run a session's Python functions. It
> installed them for the regular scalar Python UDF only, so the same logic written as a pandas UDF, a UDTF or {{applyInPandas}} received nothing — a difference users cannot
> predict from anything in the API.
> Extend the environment to every Python worker a session launches for a Python function that session supplied: scalar UDFs in all serialization modes, {{mapInPandas}} and
> {{{}mapInArrow{}}}, grouped-map, cogrouped-map, grouped-aggregate and window functions, row and Arrow UDTFs, {{applyInPandasWithState}} and {{{}transformWithState{}}}, Python data sources including the planning and streaming-reader workers, and the {{foreachBatch}} worker.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org