You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Rob Vesse (JIRA)" <ji...@apache.org> on 2018/10/02 14:03:00 UTC
[jira] [Commented] (SPARK-23153) Support application dependencies
in submission client's local file system
[ https://issues.apache.org/jira/browse/SPARK-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16635534#comment-16635534 ]
Rob Vesse commented on SPARK-23153:
-----------------------------------
[~cloud_fan][~liyinan926][~mcheah][~eje] Has there been any discussion of how to go about addressing this limitation?
In the original downstream fork there was the Resource Staging Server but that got removed to simplify upstreaming and because Spark core folks had objections to that approach. Also in our usages of it we encountered a number of performance, scalability and security issues that made it a not particularly stable approach.
There was a long dev list thread on this - https://lists.apache.org/thread.html/82b4ae9a2eb5ddeb3f7240ebf154f06f19b830f8b3120038e5d687a1@%3Cdev.spark.apache.org%3E - but no real conclusion seemed to be reached.
There are a few workarounds open to users that I can think of:
* Use the PVC support to mount a pre-created PVC that has somehow been populated with the user code
* Use the incoming pod template feature to mount arbitrary volumes that has somehow been populated with the user code
* Build custom images
All these options put the onus on users to do prep work prior to launch, I think Option 3 is currently the "recommended" workaround. Unfortunately for us that is not a viable option as our customers tend to be very security conscious and often only allow a pre-approved list of images to be run. (Ignoring the obvious fallacy of disallowing custom images while permitting the running of images that allow custom user code to execute...)
This is a blocker for me currently and I would like to contribute here but don't want to reinvent the wheel or waste effort on approaches that have already been discussed/discounted.
> Support application dependencies in submission client's local file system
> -------------------------------------------------------------------------
>
> Key: SPARK-23153
> URL: https://issues.apache.org/jira/browse/SPARK-23153
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Yinan Li
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org