You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "rickcheng (Jira)" <ji...@apache.org> on 2021/07/12 09:31:00 UTC
[jira] [Created] (SPARK-36088) 'spark.archives' does not extract
the archive file into the driver under client mode
rickcheng created SPARK-36088:
---------------------------------
Summary: 'spark.archives' does not extract the archive file into the driver under client mode
Key: SPARK-36088
URL: https://issues.apache.org/jira/browse/SPARK-36088
Project: Spark
Issue Type: Improvement
Components: Kubernetes, Spark Submit
Affects Versions: 3.1.2
Reporter: rickcheng
When running spark in the k8s cluster, there are 2 deploy modes: cluster and client. After my test, in the cluster mode, *spark.archives* can extract the archive file to the working directory of the executors and driver. But in client mode, *spark.archives* can only extract the archive file to the working directory of the executors.
However, I need *spark.archives* to send the virtual environment tar file packaged by conda to both the driver and executors under client mode (So that the executor and the driver have the same python environment).
Why *spark.archives* does not extract the archive file into the working directory of the driver under client mode?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org