You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Willi Raschkowski (Jira)" <ji...@apache.org> on 2023/11/21 10:49:00 UTC

[jira] [Commented] (SPARK-44767) Plugin API for PySpark and SparkR workers

    [ https://issues.apache.org/jira/browse/SPARK-44767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17788360#comment-17788360 ] 

Willi Raschkowski commented on SPARK-44767:
-------------------------------------------

[~gurwls223], curious what you think about this proposal. I know you're leaning towards dynamic environment selection for Spark Connect [(apache/spark#41215)|https://github.com/apache/spark/pull/41215] instead of relying on a single environment per Spark application or per host. 

At Palantir, we use conda-pack based environments with {{spark.archives}}. But that wasn't sufficient to make native library dependencies work. Internally, we implemented a {{ProcessBuilder}} plugin (using the [proposed API|https://github.com/apache/spark/pull/42440]). Among other things we use it to append the environment's {{bin/}} to the process' {{PATH}} variable or to discover Python module and non-Python binary locations outside the packaged environment.

> Plugin API for PySpark and SparkR workers
> -----------------------------------------
>
>                 Key: SPARK-44767
>                 URL: https://issues.apache.org/jira/browse/SPARK-44767
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 3.4.1
>            Reporter: Willi Raschkowski
>            Priority: Major
>              Labels: pull-request-available
>
> An API to customize Python and R workers allows for extensibility beyond what can be expressed via static configs and environment variables like, e.g., {{spark.pyspark.python}}.
> A use case for this is overriding {{PATH}} when using {{spark.archives}} with, say, conda-pack (as documented [here|https://spark.apache.org/docs/3.1.1/api/python/user_guide/python_packaging.html#using-conda]). Some packages rely on binaries. And if we want to use those packages in Spark, we need to include their binaries in the {{PATH}}.
> But we can't set the {{PATH}} via some config because 1) the environment with its binaries may be at a dynamic location (archives are unpacked on the driver [into a directory with random name|https://github.com/apache/spark/blob/5db87787d5cc1cefb51ec77e49bac7afaa46d300/core/src/main/scala/org/apache/spark/SparkFiles.scala#L33-L37]), and 2) we may not want to override the {{PATH}} that's pre-configured on the hosts.
> Other use cases unlocked by this include overriding the executable dynamically (e.g., to select a version) or forking/redirecting the worker's output stream.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org