You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2021/01/28 21:02:21 UTC
[spark] branch branch-3.1 updated: [SPARK-34281][K8S] Promote
spark.kubernetes.executor.podNamePrefix to the public conf
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new 391ba89 [SPARK-34281][K8S] Promote spark.kubernetes.executor.podNamePrefix to the public conf
391ba89 is described below
commit 391ba892c5bcf2bd1f16b708670c9a4b32f52588
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Thu Jan 28 13:01:18 2021 -0800
[SPARK-34281][K8S] Promote spark.kubernetes.executor.podNamePrefix to the public conf
### What changes were proposed in this pull request?
This PR aims to remove `internal()` from `spark.kubernetes.executor.podNamePrefix` in order to make it the configuration public.
### Why are the changes needed?
In line with K8s GA, this will allow some users control the full executor pod names officially.
This is useful when we want a custom executor pod name pattern independently from the app name.
### Does this PR introduce _any_ user-facing change?
No, this has been there since Apache Spark 2.3.0.
### How was this patch tested?
N/A.
Closes #31386 from dongjoon-hyun/SPARK-34281.
Authored-by: Dongjoon Hyun <dh...@apple.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
(cherry picked from commit 78244bafe858370deb6638b2a8d7206a195dbe52)
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
docs/running-on-kubernetes.md | 11 +++++++++++
.../src/main/scala/org/apache/spark/deploy/k8s/Config.scala | 1 -
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 93c6f94..cf5c016 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -193,6 +193,9 @@ for any reason, these pods will remain in the cluster. The executor processes sh
driver, so the executor pods should not consume compute resources (cpu and memory) in the cluster after your application
exits.
+You may use `spark.kubernetes.executor.podNamePrefix` to fully control the executor pod names.
+When this property is set, it's highly recommended to make it unique across all jobs in the same namespace.
+
### Authentication Parameters
Use the exact prefix `spark.kubernetes.authenticate` for Kubernetes authentication parameters in client mode.
@@ -874,6 +877,14 @@ See the [configuration page](configuration.html) for information on Spark config
<td>2.3.0</td>
</tr>
<tr>
+ <td><code>spark.kubernetes.executor.podNamePrefix</code></td>
+ <td>(none)</td>
+ <td>
+ Prefix to use in front of the executor pod names.
+ </td>
+ <td>2.3.0</td>
+</tr>
+<tr>
<td><code>spark.kubernetes.executor.lostCheck.maxAttempts</code></td>
<td><code>10</code></td>
<td>
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
index 487dbf5..68dcef1 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
@@ -216,7 +216,6 @@ private[spark] object Config extends Logging {
ConfigBuilder("spark.kubernetes.executor.podNamePrefix")
.doc("Prefix to use in front of the executor pod names.")
.version("2.3.0")
- .internal()
.stringConf
.createOptional
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org