You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ashangit (via GitHub)" <gi...@apache.org> on 2025/12/19 09:35:34 UTC
[PR] test [spark-kubernetes-operator]
ashangit opened a new pull request, #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, use `Draft` feature of GitHub Action.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If there is design documentation, please add the link.
3. If there is a discussion in the mailing list, please add the link.
-->
Set default configuration on the helm template in order to support high availability of the operator without requiring specific configuration from the user
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Simplify setup of HA, only need to increase the number of replicas
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
Run helm template with 1 and more replicas to check that added resources are well displayed
Deployed on a local kubernetes cluster with replicas set to 3
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2670504810
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
+ selector:
+ matchLabels:
+ app.kubernetes.io/name: spark-apache-operator
Review Comment:
Please double-check my previous comment for the existing labels.
- https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659936519
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2700759968
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-kubernetes-operator-pdb-{{ $.Release.Name }}
+ namespace: {{ .Release.Namespace }}
Review Comment:
Thank you for updating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659933139
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
+ selector:
+ matchLabels:
+ app: spark-apache-operator
Review Comment:
This looks wrong to me because `Apache Spark K8s Operator` doesn't use `app=spark-apache-operator` in the repository, @ashangit . Although I understand you are using `PodDisruptionBudget` in your environment, can we remove this from in this PR?
```
$ git grep spark-apache-operator | wc -l
0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#issuecomment-3718359901
There is a conflict. Could you rebase to the `main` branch once more to resolve the conflict, @ashangit ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#issuecomment-3762905440
cc @jiangzho because this commit will automatically create `PodDisruptionBudget` when `replicas` is greater than 1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2670493232
##########
build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml:
##########
@@ -58,6 +58,14 @@ rules:
- poddisruptionbudgets
verbs:
- '*'
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+ - apiGroups:
+ - "coordination.k8s.io"
Review Comment:
To @jiangzho , are you using this permission in our production?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659936519
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
+ selector:
+ matchLabels:
+ app: spark-apache-operator
Review Comment:
I'm wondering if this is working correctly or not because we don't use `app=spark-apache-operator` label.
```
$ kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
spark-kubernetes-operator-5f9d958469-pckdj 1/1 Running 0 78s app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator,pod-template-hash=5f9d958469,spark-role=operator
spark-kubernetes-operator-5f9d958469-xck2k 1/1 Running 0 78s app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator,pod-template-hash=5f9d958469,spark-role=operator
spark-kubernetes-operator-5f9d958469-xjmpx 1/1 Running 0 78s app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator,pod-template-hash=5f9d958469,spark-role=operator
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Add Helm configuration for spark operator HA [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2636927263
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1beta1
Review Comment:
`policy/v1` instead of `policy/v1beta1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659936519
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
+ selector:
+ matchLabels:
+ app: spark-apache-operator
Review Comment:
I'm wondering if this is working.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#issuecomment-3762903083
Merged to main.
What is your Apache JIRA ID, @ashangit ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2678302297
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
Review Comment:
This still looks inconsistent with Apache Spark convention, @ashangit .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2678303947
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
Review Comment:
Maybe, we need something like `spark-kubernetes-operator-pdb-xxx`.
Also, please double-check multiple instances across namespaces. Apache Spark Kubernetes Operator can be installed per namespace like the following official document.
https://github.com/apache/spark-kubernetes-operator/blob/main/docs/operations.md#example
```
$ k get all -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-66bc5c9577-bjd5d 1/1 Running 0 106m
kube-system pod/coredns-66bc5c9577-mzgnr 1/1 Running 0 106m
kube-system pod/etcd-docker-desktop 1/1 Running 16 106m
kube-system pod/kube-apiserver-docker-desktop 1/1 Running 16 106m
kube-system pod/kube-controller-manager-docker-desktop 1/1 Running 16 106m
kube-system pod/kube-proxy-f77lm 1/1 Running 0 106m
kube-system pod/kube-scheduler-docker-desktop 1/1 Running 16 106m
kube-system pod/storage-provisioner 1/1 Running 0 106m
kube-system pod/vpnkit-controller 1/1 Running 0 106m
us-west-1 pod/spark-kubernetes-operator-5f9d958469-xr2cz 1/1 Running 0 30s
us-west-2 pod/spark-kubernetes-operator-5f9d958469-5l6th 1/1 Running 0 15s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 106m
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 106m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 106m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 2/2 2 2 106m
us-west-1 deployment.apps/spark-kubernetes-operator 1/1 1 1 30s
us-west-2 deployment.apps/spark-kubernetes-operator 1/1 1 1 15s
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-66bc5c9577 2 2 2 106m
us-west-1 replicaset.apps/spark-kubernetes-operator-5f9d958469 1 1 1 30s
us-west-2 replicaset.apps/spark-kubernetes-operator-5f9d958469 1 1 1 15s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659936519
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
+ selector:
+ matchLabels:
+ app: spark-apache-operator
Review Comment:
I'm wondering if this is working.
```
$ kubectl get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
spark-kubernetes-operator-5f9d958469-pckdj 1/1 Running 0 78s app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator,pod-template-hash=5f9d958469,spark-role=operator
spark-kubernetes-operator-5f9d958469-xck2k 1/1 Running 0 78s app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator,pod-template-hash=5f9d958469,spark-role=operator
spark-kubernetes-operator-5f9d958469-xjmpx 1/1 Running 0 78s app.kubernetes.io/component=operator-deployment,app.kubernetes.io/name=spark-kubernetes-operator,pod-template-hash=5f9d958469,spark-role=operator
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659939148
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
Review Comment:
Could you elaborate the reason why `maxUnavailable: 1` instead of `minAvailable: 1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "ashangit (via GitHub)" <gi...@apache.org>.
ashangit commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2667836961
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
+ selector:
+ matchLabels:
+ app: spark-apache-operator
Review Comment:
Indeed this will not work. We have set specific label in our env and I badly duplicate this file without checking if it will match default labels.
I've updated it to use app.kubernetes.io/name. You can see below that it well detect pods
```
% kubectl describe poddisruptionbudget spark-apache-operator
Name: spark-apache-operator
Namespace: spark
Max unavailable: 1
Selector: app.kubernetes.io/name=spark-apache-operator
Status:
Allowed disruptions: 1
Current: 3
Desired: 2
Total: 3
Events: <none>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #437: [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily
URL: https://github.com/apache/spark-kubernetes-operator/pull/437
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2678303947
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
Review Comment:
Maybe, we need something like `spark-kubernetes-operator-pdb`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659933139
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
+ selector:
+ matchLabels:
+ app: spark-apache-operator
Review Comment:
This looks wrong to me because `Apache Spark K8s Operator` doesn't use `app=spark-apache-operator` in the repository, @ashangit . Although I understand you are using `PodDisruptionBudget` in your environment, can we remove this from in this PR?
```
$ git grep spark-apache-operator | wc -l
0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659939148
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
Review Comment:
Could you elaborate the reason why `maxUnavailable: 1` instead of `minAvailable: 1`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659940929
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
Review Comment:
Maybe, `minAvailable: 1` because `maxUnavailable: 1` may kill all operator pods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "ashangit (via GitHub)" <gi...@apache.org>.
ashangit commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2675466773
##########
build-tools/helm/spark-kubernetes-operator/templates/operator-rbac.yaml:
##########
@@ -58,6 +58,14 @@ rules:
- poddisruptionbudgets
verbs:
- '*'
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+ - apiGroups:
+ - "coordination.k8s.io"
Review Comment:
This permission is mandatory for the operator to be able to create lease when HA is enabled (ensuring only one instance run reconcile loop)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2670503333
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
+ selector:
+ matchLabels:
+ app.kubernetes.io/name: spark-apache-operator
Review Comment:
This is still wrong, @ashangit .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2678301445
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
+ selector:
+ matchLabels:
+ app.kubernetes.io/name: {{ include "spark-operator.name" . }}
Review Comment:
Thank you for the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Add Helm configuration for spark operator HA [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#issuecomment-3677561588
cc @jiangzho ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2659940929
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
+ labels:
+ {{- include "spark-operator.commonLabels" . | nindent 4 }}
+spec:
+ maxUnavailable: 1
Review Comment:
Maybe, `minAvailable: 1` because `maxUnavailable: 1` may kill all operator pods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#issuecomment-3714218521
Gentle ping, @ashangit .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2678303947
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
Review Comment:
Maybe, we need something like `spark-kubernetes-operator-pdb-xxx`.
Also, please double-check multiple instances across namespaces. Apache Spark Kubernetes Operator can be installed per namespace like the following official document.
https://github.com/apache/spark-kubernetes-operator/blob/main/docs/operations.md#example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-54896] Improve Helm templates to use Spark Operator HA feature easily [spark-kubernetes-operator]
Posted by "ashangit (via GitHub)" <gi...@apache.org>.
ashangit commented on code in PR #437:
URL: https://github.com/apache/spark-kubernetes-operator/pull/437#discussion_r2685407839
##########
build-tools/helm/spark-kubernetes-operator/templates/pdb.yaml:
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- if gt (int .Values.operatorDeployment.replicas) 1 }}
+---
+apiVersion: policy/v1
+kind: PodDisruptionBudget
+metadata:
+ name: spark-apache-operator
Review Comment:
A pdb can only select for pod in its namespace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org