You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Santosh Pingale (Jira)" <ji...@apache.org> on 2023/01/24 14:26:00 UTC
[jira] [Updated] (SPARK-42170) Files added to the spark-submit command with master K8s and deploy mode cluster, end up in a non deterministic location inside the driver.
[ https://issues.apache.org/jira/browse/SPARK-42170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Santosh Pingale updated SPARK-42170:
------------------------------------
Description:
Files added to the spark-submit command with master K8s and deploy mode cluster, end up in a non deterministic location inside the driver.
eg:
{{spark-submit --files myfile --master k8s.. --deploy-mode cluster` will upload the files to /tmp/spark-uuid/myfile}}
The issue happens because [Utils.createTempDir()|https://github.com/apache/spark/blob/v3.3.1/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L344] creates a directory with a uuid in the directory name. This issue does not affect the --archives option, because we `unarchive` the archives into the destination directory which is relative to the working dir. This bug affects file access pre & post app creation. For example if we distribute python dependencies with pex, we need to use --files to attach the pex file and change the spark.pyspark.python to point to this file. But the file location can not be determined before submitting the app. On the other hand, after the app is created, referencing the files without using `SparkFiles.get` also does not work
was:
Files added to the spark-submit command with master K8s and deploy mode cluster, end up in a non deterministic location inside the driver.
eg:
{{spark-submit --files myfile --master k8s.. --deploy-mode cluster` will upload the files to /tmp/spark-uuid/myfile}}
The issue happens because [Utils.createTempDir()|https://github.com/apache/spark/blob/v3.3.1/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L344] creates a directory with a uuid in the directory name. This issue does not affect the `--archives` option, because we `unarchive` the archives into the destination directory which is relative to the working dir. This bug affects file access pre & post app creation. For example if we distribute python dependencies with pex, we need to use `--files` to attach the pex file and change the spark.pyspark.python to point to this file. But the file location can not be determined before submitting the app. On the other hand, after the app is created, referencing the files without using `SparkFiles.get` also does not work
> Files added to the spark-submit command with master K8s and deploy mode cluster, end up in a non deterministic location inside the driver.
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-42170
> URL: https://issues.apache.org/jira/browse/SPARK-42170
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Spark Submit
> Affects Versions: 3.3.0, 3.2.2
> Reporter: Santosh Pingale
> Priority: Major
>
> Files added to the spark-submit command with master K8s and deploy mode cluster, end up in a non deterministic location inside the driver.
> eg:
> {{spark-submit --files myfile --master k8s.. --deploy-mode cluster` will upload the files to /tmp/spark-uuid/myfile}}
> The issue happens because [Utils.createTempDir()|https://github.com/apache/spark/blob/v3.3.1/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L344] creates a directory with a uuid in the directory name. This issue does not affect the --archives option, because we `unarchive` the archives into the destination directory which is relative to the working dir. This bug affects file access pre & post app creation. For example if we distribute python dependencies with pex, we need to use --files to attach the pex file and change the spark.pyspark.python to point to this file. But the file location can not be determined before submitting the app. On the other hand, after the app is created, referencing the files without using `SparkFiles.get` also does not work
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org