You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/07/25 00:31:08 UTC

(spark) branch branch-4.x updated: [SPARK-58334][K8S][DOC] Introduce Apache Spark K8s Operator in `running-on-kubernetes.md`

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 954b8a831e32 [SPARK-58334][K8S][DOC] Introduce Apache Spark K8s Operator in `running-on-kubernetes.md`
954b8a831e32 is described below

commit 954b8a831e32c9cb6215a6a8c4bb12a0cb71a6cb
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Fri Jul 24 17:30:46 2026 -0700

    [SPARK-58334][K8S][DOC] Introduce Apache Spark K8s Operator in `running-on-kubernetes.md`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to introduce `Apache Spark Kubernetes Operator` in `running-on-kubernetes.md` by adding a short section after `How it works` section.
    
    - What the operator is: a separate Apache Spark project that deploys and manages Spark
      workloads declaratively via `SparkApp` and `SparkCluster` custom resources.
    - Its relationship with the `spark-submit` based submission covered by this document:
      the two approaches are complementary and the operator runs on top of the same native
      Kubernetes scheduler backend.
    - Links to the operator repository and examples, consistent with the existing links in
      `docs/index.md`.
    
    ### Why are the changes needed?
    
    `Apache Spark Kubernetes Operator` v1.0.0 is going to be released Tomorrow. Currently, it is only mentioned in `docs/index.md`. Users who land directly on `running-on-kubernetes.md`, the main documentation page for running Spark on Kubernetes, have no way to discover the operator. This closes that discoverability gap while keeping the detailed operator usage out of scope (deferred to the operator repository).
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. This is a documentation-only change.
    
    ### How was this patch tested?
    
    Manually check.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Fable 5
    
    Closes #57512 from dongjoon-hyun/SPARK-58334.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit 0700340fa6fc54f1f625b49503c205eab44394eb)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 docs/running-on-kubernetes.md | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index b88a313861ca..e2b88fd9f2a4 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -80,6 +80,20 @@ driver and executor pods on a subset of available nodes through a [node selector
 using the configuration property for it. It will be possible to use more advanced
 scheduling hints like [node/pod affinities](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity) in a future release.
 
+# Spark Kubernetes Operator
+
+In addition to the `spark-submit` based submission described in this document, users can deploy and
+manage Spark workloads declaratively via [operator patterns](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
+by using [Apache Spark Kubernetes Operator](https://github.com/apache/spark-kubernetes-operator),
+a separate Apache Spark project. The operator provides two custom resources:
+
+* [SparkApp](https://github.com/apache/spark-kubernetes-operator/blob/main/examples/pi-python.yaml): deploy Spark apps on top of Kubernetes
+* [SparkCluster](https://github.com/apache/spark-kubernetes-operator/blob/main/examples/cluster.yaml): deploy Spark clusters on top of Kubernetes
+
+The two approaches are complementary because the operator runs Spark applications on top of the same
+native Kubernetes scheduler backend described in this document. Please refer to the operator
+repository for its detailed documentation and usage.
+
 # Submitting Applications to Kubernetes
 
 ## Docker Images


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org