You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Oleg Frenkel (JIRA)" <ji...@apache.org> on 2019/01/30 20:26:00 UTC
[jira] [Created] (SPARK-26789) [k8s] pyspark needs to upload local
resources to driver and executor pods
Oleg Frenkel created SPARK-26789:
------------------------------------
Summary: [k8s] pyspark needs to upload local resources to driver and executor pods
Key: SPARK-26789
URL: https://issues.apache.org/jira/browse/SPARK-26789
Project: Spark
Issue Type: New Feature
Components: Kubernetes, PySpark
Affects Versions: 2.4.0
Reporter: Oleg Frenkel
Kubernetes support provided with [https://github.com/apache-spark-on-k8s/spark] allows local dependencies to be used with cluster deployment mode. Specifically, the Resource Staging Server is used in order to upload local dependencies to Kubernetes so that driver and executor pods can download these dependencies. It looks that Spark 2.4.0 release does not support local dependencies.
For example, the following command is expected to automatically upload pi.py from local machine to the Kubernetes cluster and make it available for both driver and executor pods:
{{bin/spark-submit --conf spark.app.name=example.python.pi --master k8s://http://127.0.0.1:8001 --deploy-mode cluster --conf spark.kubernetes.container.image=spark-py:spark-2.4.0 ./examples/src/main/python/pi.py}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org