[spark] branch master updated: [SPARK-38561][K8S][DOCS] Add doc for `Customized Kubernetes Schedulers`

[ho...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

holden pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 4ff40c1  [SPARK-38561][K8S][DOCS] Add doc for `Customized Kubernetes Schedulers`
4ff40c1 is described below

commit 4ff40c10f02f6e0735ce6554f7338489d8555bce
Author: Yikun Jiang <yi...@gmail.com>
AuthorDate: Wed Mar 16 11:12:54 2022 -0700

    [SPARK-38561][K8S][DOCS] Add doc for `Customized Kubernetes Schedulers`
    
    ### What changes were proposed in this pull request?
    This is PR to doc for basic framework capability for Customized Kubernetes Schedulers.
    
    ### Why are the changes needed?
    Guide user how to use spark on kubernetes custom scheduler
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    CI passed
    
    Closes #35869 from Yikun/SPARK-38561.
    
    Authored-by: Yikun Jiang <yi...@gmail.com>
    Signed-off-by: Holden Karau <hk...@netflix.com>
---
 docs/running-on-kubernetes.md | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index de37e22..d1b2fcd 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1713,6 +1713,25 @@ spec:
     image: will-be-overwritten
 ```
 
+#### Customized Kubernetes Schedulers for Spark on Kubernetes
+
+Spark allows users to specify a custom Kubernetes schedulers.
+
+1. Specify scheduler name.
+
+   Users can specify a custom scheduler using <code>spark.kubernetes.scheduler.name</code> or
+   <code>spark.kubernetes.{driver/executor}.scheduler.name</code> configuration.
+
+2. Specify scheduler related configurations.
+
+   To configure the custom scheduler the user can use [Pod templates](#pod-template), add labels (<code>spark.kubernetes.{driver,executor}.label.*</code>)  and/or annotations (<code>spark.kubernetes.{driver/executor}.annotation.*</code>).
+
+3. Specify scheduler feature step.
+
+   Users may also consider to use <code>spark.kubernetes.{driver/executor}.pod.featureSteps</code> to support more complex requirements, including but not limited to:
+  - Create additional Kubernetes custom resources for driver/executor scheduling.
+  - Set scheduler hints according to configuration or existing Pod info dynamically.
+
 ### Stage Level Scheduling Overview
 
 Stage level scheduling is supported on Kubernetes when dynamic allocation is enabled. This also requires <code>spark.dynamicAllocation.shuffleTracking.enabled</code> to be enabled since Kubernetes doesn't support an external shuffle service at this time. The order in which containers for different profiles is requested from Kubernetes is not guaranteed. Note that since dynamic allocation on Kubernetes requires the shuffle tracking feature, this means that executors from previous stages t [...]

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org