You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by abhisrao <gi...@git.apache.org> on 2018/09/24 11:15:59 UTC
[GitHub] spark pull request #22536: [SPARK-25500][K8s]Specify configmap and secrets i...
GitHub user abhisrao opened a pull request:
https://github.com/apache/spark/pull/22536
[SPARK-25500][K8s]Specify configmap and secrets in Spark driver and executor pods in Kubernetes
## What changes were proposed in this pull request?
This PR introduces support for specifying configmap as spark-configuration.
Using PR #22146, the above functionality can be achieved by passing template file. However, for spark properties (like log4j.properties, fairscheduler.xml and metrics.properties), we are proposing this approach as this is native to other configuration options specifications in spark.
The configmaps has to be pre-created before using this as spark configuration.
(Please fill in changes proposed in this fix)
Add the following parameters while doing spark-submit
spark.kubernetes.driver.volumes.configMap.<volume-name>.mount.path=<mount location inside pod>
spark.kubernetes.driver.volumes.configMap.<volume-name>.options.name=<configmap name>
Using these properties, mount the configmap to spark-driver and spark-executor pods.
Changes are required in the `KubernetesVolumeSpec` and `MountVolumesFeatureStep` .
## How was this patch tested?
Manual testing on a k8s cluster. Unit tests added to `KubernetesVolumeUtilsSuite` and `MountVolumesFeatureStepSuite`
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/abhisrao/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22536.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22536
----
commit 54fd540680e26a8a40dff31be42f6605f0707687
Author: RAO,ABHISHEK (ABHISHEK) <ab...@...>
Date: 2018-09-23T16:22:17Z
First Commit
commit 3258ff63a1aaec2107518c73cd207e5400a1eca5
Author: RAO,ABHISHEK (ABHISHEK) <ab...@...>
Date: 2018-09-23T17:04:49Z
Documentation update
commit 606c13ea08e6c5b341e78ae99cf9ca2dd57eaa75
Author: RAO,ABHISHEK (ABHISHEK) <ab...@...>
Date: 2018-09-24T09:39:37Z
Removed changes for secret as this is already supported since Spark 2.3
commit 9f48160cd64255c4b90eadfe1db8aacbb7fd7566
Author: RAO,ABHISHEK (ABHISHEK) <ab...@...>
Date: 2018-09-24T09:39:59Z
Removed changes for secret as this is already supported since Spark 2.3
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22536: [SPARK-25500][K8s]Specify configmap and secrets in Spark...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22536
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22536: [SPARK-25500][K8s]Specify configmap and secrets in Spark...
Posted by abhisrao <gi...@git.apache.org>.
Github user abhisrao commented on the issue:
https://github.com/apache/spark/pull/22536
ok. Thanks. We'll use the templates way to mount configmap. Hence closing the PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22536: [SPARK-25500][K8s]Specify configmap and secrets in Spark...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22536
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22536: [SPARK-25500][K8s]Specify configmap and secrets i...
Posted by abhisrao <gi...@git.apache.org>.
Github user abhisrao closed the pull request at:
https://github.com/apache/spark/pull/22536
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22536: [SPARK-25500][K8s]Specify configmap and secrets in Spark...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/22536
As I said in the JIRA, we no longer add new configuration options for pod customization. Please consider using the changes in https://github.com/apache/spark/pull/22146 for mounting ConfigMaps.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22536: [SPARK-25500][K8s]Specify configmap and secrets in Spark...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22536
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org