You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2021/12/01 06:50:34 UTC
[spark] branch master updated: [SPARK-37480][K8S][DOC] Sync Kubernetes configuration to latest in running-on-k8s.md
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f7be024 [SPARK-37480][K8S][DOC] Sync Kubernetes configuration to latest in running-on-k8s.md
f7be024 is described below
commit f7be0248f84a67d4fa839dc2d6e496c5a7d96830
Author: Yikun Jiang <yi...@gmail.com>
AuthorDate: Tue Nov 30 22:49:32 2021 -0800
[SPARK-37480][K8S][DOC] Sync Kubernetes configuration to latest in running-on-k8s.md
### What changes were proposed in this pull request?
Sync Kubernetes configurations to latest in doc
### Why are the changes needed?
Configurations in docs/running-on-kubernetes.md are not uptodate
### Does this PR introduce _any_ user-facing change?
No, docs only
### How was this patch tested?
CI passed
Closes #34734 from Yikun/SPARK-37480.
Authored-by: Yikun Jiang <yi...@gmail.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
docs/running-on-kubernetes.md | 175 ++++++++++++++++++++++++++++++++++++++++--
1 file changed, 170 insertions(+), 5 deletions(-)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index d32861b..58d6dda 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -592,6 +592,7 @@ See the [configuration page](configuration.html) for information on Spark config
<td><code>IfNotPresent</code></td>
<td>
Container image pull policy used when pulling images within Kubernetes.
+ Valid values are <code>Always</code>, <code>Never</code>, and <code>IfNotPresent</code>.
</td>
<td>2.3.0</td>
</tr>
@@ -780,6 +781,15 @@ See the [configuration page](configuration.html) for information on Spark config
<td>2.3.0</td>
</tr>
<tr>
+ <td><code>spark.kubernetes.authenticate.executor.serviceAccountName</code></td>
+ <td><code>(value of spark.kubernetes.authenticate.driver.serviceAccountName)</code></td>
+ <td>
+ Service account that is used when running the executor pod.
+ If this parameter is not setup, the fallback logic will use the driver's service account.
+ </td>
+ <td>3.1.0</td>
+</tr>
+<tr>
<td><code>spark.kubernetes.authenticate.caCertFile</code></td>
<td>(none)</td>
<td>
@@ -925,6 +935,14 @@ See the [configuration page](configuration.html) for information on Spark config
<td>2.3.0</td>
</tr>
<tr>
+ <td><code>spark.kubernetes.executor.apiPollingInterval</code></td>
+ <td><code>30s</code></td>
+ <td>
+ Interval between polls against the Kubernetes API server to inspect the state of executors.
+ </td>
+ <td>2.4.0</td>
+</tr>
+<tr>
<td><code>spark.kubernetes.driver.request.cores</code></td>
<td>(none)</td>
<td>
@@ -1232,7 +1250,7 @@ See the [configuration page](configuration.html) for information on Spark config
</tr>
<tr>
<td><code>spark.kubernetes.executor.checkAllContainers</code></td>
- <td>false</td>
+ <td><code>false</code></td>
<td>
Specify whether executor pods should be check all containers (including sidecars) or only the executor container when determining the pod status.
</td>
@@ -1240,7 +1258,7 @@ See the [configuration page](configuration.html) for information on Spark config
</tr>
<tr>
<td><code>spark.kubernetes.submission.connectionTimeout</code></td>
- <td>10000</td>
+ <td><code>10000</code></td>
<td>
Connection timeout in milliseconds for the kubernetes client to use for starting the driver.
</td>
@@ -1248,7 +1266,7 @@ See the [configuration page](configuration.html) for information on Spark config
</tr>
<tr>
<td><code>spark.kubernetes.submission.requestTimeout</code></td>
- <td>10000</td>
+ <td><code>10000</code></td>
<td>
Request timeout in milliseconds for the kubernetes client to use for starting the driver.
</td>
@@ -1256,7 +1274,7 @@ See the [configuration page](configuration.html) for information on Spark config
</tr>
<tr>
<td><code>spark.kubernetes.driver.connectionTimeout</code></td>
- <td>10000</td>
+ <td><code>10000</code></td>
<td>
Connection timeout in milliseconds for the kubernetes client in driver to use when requesting executors.
</td>
@@ -1264,7 +1282,7 @@ See the [configuration page](configuration.html) for information on Spark config
</tr>
<tr>
<td><code>spark.kubernetes.driver.requestTimeout</code></td>
- <td>10000</td>
+ <td><code>10000</code></td>
<td>
Request timeout in milliseconds for the kubernetes client in driver to use when requesting executors.
</td>
@@ -1279,6 +1297,14 @@ See the [configuration page](configuration.html) for information on Spark config
<td>3.0.0</td>
</tr>
<tr>
+ <td><code>spark.kubernetes.dynamicAllocation.deleteGracePeriod</code></td>
+ <td><code>5s</code></td>
+ <td>
+ How long to wait for executors to shut down gracefully before a forceful kill.
+ </td>
+ <td>3.0.0</td>
+</tr>
+<tr>
<td><code>spark.kubernetes.file.upload.path</code></td>
<td>(none)</td>
<td>
@@ -1322,6 +1348,145 @@ See the [configuration page](configuration.html) for information on Spark config
</td>
<td>3.3.0</td>
</tr>
+<tr>
+ <td><code>spark.kubernetes.configMap.maxSize</code></td>
+ <td><code>1572864</code></td>
+ <td>
+ Max size limit for a config map.
+ This is configurable as per <a href="https://etcd.io/docs/latest/dev-guide/limit/">limit</a> on k8s server end.
+ </td>
+ <td>3.1.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.executor.missingPodDetectDelta</code></td>
+ <td><code>30s</code></td>
+ <td>
+ When a registered executor's POD is missing from the Kubernetes API server's polled
+ list of PODs then this delta time is taken as the accepted time difference between the
+ registration time and the time of the polling. After this time the POD is considered
+ missing from the cluster and the executor will be removed.
+ </td>
+ <td>3.1.1</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.decommission.script</code></td>
+ <td><code>/opt/decom.sh</code></td>
+ <td>
+ The location of the script to use for graceful decommissioning.
+ </td>
+ <td>3.2.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.driver.service.deleteOnTermination</code></td>
+ <td><code>true</code></td>
+ <td>
+ If true, driver service will be deleted on Spark application termination. If false, it will be cleaned up when the driver pod is deletion.
+ </td>
+ <td>3.2.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.driver.ownPersistentVolumeClaim</code></td>
+ <td><code>false</code></td>
+ <td>
+ If true, driver pod becomes the owner of on-demand persistent volume claims instead of the executor pods
+ </td>
+ <td>3.2.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.driver.reusePersistentVolumeClaim</code></td>
+ <td><code>false</code></td>
+ <td>
+ If true, driver pod tries to reuse driver-owned on-demand persistent volume claims
+ of the deleted executor pods if exists. This can be useful to reduce executor pod
+ creation delay by skipping persistent volume creations. Note that a pod in
+ `Terminating` pod status is not a deleted pod by definition and its resources
+ including persistent volume claims are not reusable yet. Spark will create new
+ persistent volume claims when there exists no reusable one. In other words, the total
+ number of persistent volume claims can be larger than the number of running executors
+ sometimes. This config requires <code>spark.kubernetes.driver.ownPersistentVolumeClaim=true.</code>
+ </td>
+ <td>3.2.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.executor.disableConfigMap</code></td>
+ <td><code>false</code></td>
+ <td>
+ If true, disable ConfigMap creation for executors.
+ </td>
+ <td>3.2.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.driver.pod.featureSteps</code></td>
+ <td>(none)</td>
+ <td>
+ Class names of an extra driver pod feature step implementing
+ `KubernetesFeatureConfigStep`. This is a developer API. Comma separated.
+ Runs after all of Spark internal feature steps.
+ </td>
+ <td>3.2.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.executor.pod.featureSteps</code></td>
+ <td>(none)</td>
+ <td>
+ Class names of an extra executor pod feature step implementing
+ `KubernetesFeatureConfigStep`. This is a developer API. Comma separated.
+ Runs after all of Spark internal feature steps.
+ </td>
+ <td>3.2.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.allocation.maxPendingPods</code></td>
+ <td><code>Int.MaxValue</code></td>
+ <td>
+ Maximum number of pending PODs allowed during executor allocation for this
+ application. Those newly requested executors which are unknown by Kubernetes yet are
+ also counted into this limit as they will change into pending PODs by time.
+ This limit is independent from the resource profiles as it limits the sum of all
+ allocation for all the used resource profiles.
+ </td>
+ <td>3.2.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.allocation.pods.allocator</code></td>
+ <td><code>direct</code></td>
+ <td>
+ Allocator to use for pods. Possible values are <code>direct</code> (the default)
+ and <code>statefulset</code>, or a full class name of a class implementing `AbstractPodsAllocator`.
+ Future version may add Job or replicaset. This is a developer API and may change
+ or be removed at anytime.
+ </td>
+ <td>3.3.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.allocation.executor.timeout</code></td>
+ <td><code>600s</code></td>
+ <td>
+ Time to wait before a newly created executor POD request, which does not reached
+ the POD pending state yet, considered timedout and will be deleted.
+ </td>
+ <td>3.1.0</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.allocation.driver.readinessTimeout</code></td>
+ <td><code>1s</code></td>
+ <td>
+ Time to wait for driver pod to get ready before creating executor pods. This wait
+ only happens on application start. If timeout happens, executor pods will still be
+ created.
+ </td>
+ <td>3.1.3</td>
+</tr>
+<tr>
+ <td><code>spark.kubernetes.executor.enablePollingWithResourceVersion</code></td>
+ <td><code>false</code></td>
+ <td>
+ If true, `resourceVersion` is set with `0` during invoking pod listing APIs
+ in order to allow API Server-side caching. This should be used carefully.
+ </td>
+ <td>3.3.0</td>
+</tr>
+
</table>
#### Pod template properties
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org