You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Dongjoon Hyun <do...@apache.org> on 2026/07/26 22:46:43 UTC
[ANNOUNCE] Apache Spark K8s Operator 1.0.0 released
Hi all,
On behalf of the Apache Spark PMC, I am proud to announce the release
of Apache Spark K8s Operator 1.0.0!
This is the first stable release of the Apache Spark K8s Operator, an
official subproject of Apache Spark. It marks a major milestone for the
Spark community: for the first time, the Apache Spark project itself
provides a complete, community-driven, vendor-neutral standard for
running Spark on Kubernetes via the Operator pattern.
Built on the Java Operator SDK and the Fabric8 Kubernetes client, the
operator manages the full lifecycle of Spark workloads through two
Kubernetes custom resources:
- SparkApplication: submit and manage individual Spark applications
(Java/Scala, PySpark, and SQL) with a rich state machine covering
submission, execution, failure handling, retries, and resource
cleanup.
- SparkCluster: provision and operate standalone Spark clusters,
including Spark Connect servers, Spark History Server, and
Spark Thrift Server deployments, with StatefulSet-based masters
and workers and Horizontal Pod Autoscaler support.
Highlights of 1.0.0:
- Supports Apache Spark 4.0 and above and
recent Kubernetes versions (1.34+ recommended).
- First-class Spark Connect support for next-generation
client-server Spark workloads.
- Helm-chart based installation, published on Artifact Hub, with
namespace-scoped RBAC and multi-namespace watching.
- Gateway API (HTTPRoute/GRPCRoute) integration for exposing
driver UIs and Spark Connect endpoints.
- Production-grade operations: metrics, health probes, leader
election, and extensive end-to-end test coverage via Chainsaw.
Just as importantly, 1.0.0 was designed and verified as the hub of the
broader Apache Spark ecosystem. Out of the box, it ships with working
examples and e2e coverage for:
- Apache Iceberg table format with Spark Connect
- Apache Hadoop-compatible storage and S3-compatible object stores
- Apache DataFusion Comet and Apache Gluten native accelerators
- Apache Celeborn remote shuffle service
- Apache YuniKorn and Volcano batch schedulers
We believe this release establishes the standard way to run the entire
Spark ecosystem on Kubernetes, and we are proud of what the community
has built together.
To get started:
helm repo add spark https://apache.github.io/spark-kubernetes-operator
helm repo update
helm install spark spark/spark-kubernetes-operator
Release Note:
- https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0
- https://github.com/apache/spark-kubernetes-operator/milestone/3
- https://s.apache.org/spark-kubernetes-operator-1.0.0
Published Docker Image:
- apache/spark-kubernetes-operator:1.0.0
Useful links:
- Download:
https://downloads.apache.org/spark/spark-kubernetes-operator-1.0.0/
- Website: https://s.apache.org/spark-kubernetes-operator/
- Source: https://github.com/apache/spark-kubernetes-operator
- Artifact Hub:
https://artifacthub.io/packages/search?repo=spark-kubernetes-operator
- Documentation:
https://github.com/apache/spark-kubernetes-operator/tree/main/docs
We would like to thank all the contributors, reviewers, and users who
made this first release possible. This release is the result of the
Apache Spark community's collaborative effort, and we look forward to
your feedback and contributions.
Regards,
Dongjoon Hyun
on behalf of the Apache Spark PMC
Re: [ANNOUNCE] Apache Spark K8s Operator 1.0.0 released
Posted by Uroš Bojanić <ur...@apache.org>.
This is great! Thank you Dongjoon 🎉
On 2026/07/26 22:46:43 Dongjoon Hyun wrote:
> Hi all,
>
> On behalf of the Apache Spark PMC, I am proud to announce the release
> of Apache Spark K8s Operator 1.0.0!
>
> This is the first stable release of the Apache Spark K8s Operator, an
> official subproject of Apache Spark. It marks a major milestone for the
> Spark community: for the first time, the Apache Spark project itself
> provides a complete, community-driven, vendor-neutral standard for
> running Spark on Kubernetes via the Operator pattern.
>
> Built on the Java Operator SDK and the Fabric8 Kubernetes client, the
> operator manages the full lifecycle of Spark workloads through two
> Kubernetes custom resources:
>
> - SparkApplication: submit and manage individual Spark applications
> (Java/Scala, PySpark, and SQL) with a rich state machine covering
> submission, execution, failure handling, retries, and resource
> cleanup.
> - SparkCluster: provision and operate standalone Spark clusters,
> including Spark Connect servers, Spark History Server, and
> Spark Thrift Server deployments, with StatefulSet-based masters
> and workers and Horizontal Pod Autoscaler support.
>
> Highlights of 1.0.0:
>
> - Supports Apache Spark 4.0 and above and
> recent Kubernetes versions (1.34+ recommended).
> - First-class Spark Connect support for next-generation
> client-server Spark workloads.
> - Helm-chart based installation, published on Artifact Hub, with
> namespace-scoped RBAC and multi-namespace watching.
> - Gateway API (HTTPRoute/GRPCRoute) integration for exposing
> driver UIs and Spark Connect endpoints.
> - Production-grade operations: metrics, health probes, leader
> election, and extensive end-to-end test coverage via Chainsaw.
>
> Just as importantly, 1.0.0 was designed and verified as the hub of the
> broader Apache Spark ecosystem. Out of the box, it ships with working
> examples and e2e coverage for:
>
> - Apache Iceberg table format with Spark Connect
> - Apache Hadoop-compatible storage and S3-compatible object stores
> - Apache DataFusion Comet and Apache Gluten native accelerators
> - Apache Celeborn remote shuffle service
> - Apache YuniKorn and Volcano batch schedulers
>
> We believe this release establishes the standard way to run the entire
> Spark ecosystem on Kubernetes, and we are proud of what the community
> has built together.
>
> To get started:
>
> helm repo add spark https://apache.github.io/spark-kubernetes-operator
> helm repo update
> helm install spark spark/spark-kubernetes-operator
>
> Release Note:
>
> - https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0
> - https://github.com/apache/spark-kubernetes-operator/milestone/3
> - https://s.apache.org/spark-kubernetes-operator-1.0.0
>
> Published Docker Image:
>
> - apache/spark-kubernetes-operator:1.0.0
>
> Useful links:
>
> - Download:
> https://downloads.apache.org/spark/spark-kubernetes-operator-1.0.0/
> - Website: https://s.apache.org/spark-kubernetes-operator/
> - Source: https://github.com/apache/spark-kubernetes-operator
> - Artifact Hub:
> https://artifacthub.io/packages/search?repo=spark-kubernetes-operator
> - Documentation:
> https://github.com/apache/spark-kubernetes-operator/tree/main/docs
>
> We would like to thank all the contributors, reviewers, and users who
> made this first release possible. This release is the result of the
> Apache Spark community's collaborative effort, and we look forward to
> your feedback and contributions.
>
> Regards,
> Dongjoon Hyun
> on behalf of the Apache Spark PMC
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [ANNOUNCE] Apache Spark K8s Operator 1.0.0 released
Posted by Zhou Jiang <zh...@gmail.com>.
Great to see the 1.0 release out!
Thanks Dongjoon for driving the release, and thanks everyone who
contributed along the way!
*Zhou JIANG*
On Sun, Jul 26, 2026 at 15:47 Dongjoon Hyun <do...@apache.org> wrote:
> Hi all,
>
> On behalf of the Apache Spark PMC, I am proud to announce the release
> of Apache Spark K8s Operator 1.0.0!
>
> This is the first stable release of the Apache Spark K8s Operator, an
> official subproject of Apache Spark. It marks a major milestone for the
> Spark community: for the first time, the Apache Spark project itself
> provides a complete, community-driven, vendor-neutral standard for
> running Spark on Kubernetes via the Operator pattern.
>
> Built on the Java Operator SDK and the Fabric8 Kubernetes client, the
> operator manages the full lifecycle of Spark workloads through two
> Kubernetes custom resources:
>
> - SparkApplication: submit and manage individual Spark applications
> (Java/Scala, PySpark, and SQL) with a rich state machine covering
> submission, execution, failure handling, retries, and resource
> cleanup.
> - SparkCluster: provision and operate standalone Spark clusters,
> including Spark Connect servers, Spark History Server, and
> Spark Thrift Server deployments, with StatefulSet-based masters
> and workers and Horizontal Pod Autoscaler support.
>
> Highlights of 1.0.0:
>
> - Supports Apache Spark 4.0 and above and
> recent Kubernetes versions (1.34+ recommended).
> - First-class Spark Connect support for next-generation
> client-server Spark workloads.
> - Helm-chart based installation, published on Artifact Hub, with
> namespace-scoped RBAC and multi-namespace watching.
> - Gateway API (HTTPRoute/GRPCRoute) integration for exposing
> driver UIs and Spark Connect endpoints.
> - Production-grade operations: metrics, health probes, leader
> election, and extensive end-to-end test coverage via Chainsaw.
>
> Just as importantly, 1.0.0 was designed and verified as the hub of the
> broader Apache Spark ecosystem. Out of the box, it ships with working
> examples and e2e coverage for:
>
> - Apache Iceberg table format with Spark Connect
> - Apache Hadoop-compatible storage and S3-compatible object stores
> - Apache DataFusion Comet and Apache Gluten native accelerators
> - Apache Celeborn remote shuffle service
> - Apache YuniKorn and Volcano batch schedulers
>
> We believe this release establishes the standard way to run the entire
> Spark ecosystem on Kubernetes, and we are proud of what the community
> has built together.
>
> To get started:
>
> helm repo add spark https://apache.github.io/spark-kubernetes-operator
> helm repo update
> helm install spark spark/spark-kubernetes-operator
>
> Release Note:
>
> - https://github.com/apache/spark-kubernetes-operator/releases/tag/1.0.0
> - https://github.com/apache/spark-kubernetes-operator/milestone/3
> - https://s.apache.org/spark-kubernetes-operator-1.0.0
>
> Published Docker Image:
>
> - apache/spark-kubernetes-operator:1.0.0
>
> Useful links:
>
> - Download:
> https://downloads.apache.org/spark/spark-kubernetes-operator-1.0.0/
> - Website: https://s.apache.org/spark-kubernetes-operator/
> - Source: https://github.com/apache/spark-kubernetes-operator
> - Artifact Hub:
> https://artifacthub.io/packages/search?repo=spark-kubernetes-operator
> - Documentation:
> https://github.com/apache/spark-kubernetes-operator/tree/main/docs
>
> We would like to thank all the contributors, reviewers, and users who
> made this first release possible. This release is the result of the
> Apache Spark community's collaborative effort, and we look forward to
> your feedback and contributions.
>
> Regards,
> Dongjoon Hyun
> on behalf of the Apache Spark PMC
>