You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Stavros Kontopoulos (Jira)" <ji...@apache.org> on 2021/09/17 18:22:00 UTC
[jira] [Comment Edited] (SPARK-23153) Support application
dependencies in submission client's local file system
[ https://issues.apache.org/jira/browse/SPARK-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17416846#comment-17416846 ]
Stavros Kontopoulos edited comment on SPARK-23153 at 9/17/21, 6:21 PM:
-----------------------------------------------------------------------
[~xuzhoyin] sorry for the late reply, the local scheme in the past meant local in the container, had a different meaning (https://github.com/apache/spark/pull/21378). So this was intentional. Not sure the status now.
was (Author: skonto):
[~xuzhoyin] sorry for the late reply, the local scheme in the past meant local in the container, had a different meaning (https://github.com/apache/spark/pull/21378). Not sure the status now.
> Support application dependencies in submission client's local file system
> -------------------------------------------------------------------------
>
> Key: SPARK-23153
> URL: https://issues.apache.org/jira/browse/SPARK-23153
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes, Spark Core
> Affects Versions: 2.4.0
> Reporter: Yinan Li
> Assignee: Stavros Kontopoulos
> Priority: Major
> Fix For: 3.0.0
>
>
> Currently local dependencies are not supported with Spark on K8S i.e. if the user has code or dependencies only on the client where they run {{spark-submit}} then the current implementation has no way to make those visible to the Spark application running inside the K8S pods that get launched. This limits users to only running applications where the code and dependencies are either baked into the Docker images used or where those are available via some external and globally accessible file system e.g. HDFS which are not viable options for many users and environments
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org