You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/07/13 11:00:37 UTC
[GitHub] [pulsar-helm-chart] mkozioro opened a new issue #39: Bug in podAntiAffinity
mkozioro opened a new issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39
**Describe the bug**
I've noticed, that multiple zookeeper pods started on same kubernetes node. It looks like issue with labels and podAntiAffinity settings.
Example using zookeeper
koziorow@pl1lxl-108450:~/projects/LNS-DevEnv/repo/batch-service-infra/terraform-apache-pulsar$ kubectl describe pod pulsar-zookeeper-0
Name: pulsar-zookeeper-0
Namespace: default
Priority: 0
Node: aks-default-12797261-vmss000001/10.240.0.115
Start Time: Mon, 13 Jul 2020 12:22:02 +0200
Labels: **app=pulsar**
cluster=pulsar
component=zookeeper
controller-revision-hash=pulsar-zookeeper-7d87bf67f9
release=pulsar
statefulset.kubernetes.io/pod-name=pulsar-zookeeper-0
In chart:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
**- "{{ template "pulsar.name" . }}-{{ .Values.zookeeper.component }}"**
- key: "release"
operator: In
values:
- {{ .Release.Name }}
- key: "component"
operator: In
values:
- {{ .Values.zookeeper.component }}
topologyKey: "kubernetes.io/hostname"
**Expected behavior**
Zookeeper pods should start on different nodes. Problem exists also in other components.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] rvashishth commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
rvashishth commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-662924558
@mkozioro which cloud are you using, I could see the behavior as you mentioned on Azure AKS.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] rvashishth commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
rvashishth commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-660312628
@mkozioro I tried the anti-affinity on AKS, but I can see pods are scheduled on different AKS nodes as expected. I have tested with 3 node-pool, 3 zookeeper pods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] mkozioro commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
mkozioro commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-660357711
@rvashishth Could you try with 3 node pool and 5 zookeepers? If affinity works OK, 2 new nodes should be always created. When I was testing couple days ago, I've got 2 zookeepers running on same node.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] rvashishth commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
rvashishth commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-658352260
I was about to create an issue for standard helm lable and annotations. These will be important to migrate this chart from helm 2 to helm 3.
We should also upgrade the chart api to version 3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] sijie closed issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
sijie closed issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] sijie commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-673232607
> I can open a PR with this fix if you would like.
+1
> Should we set the affinity to preferredDuringSchedulingIgnoredDuringExecution?
Can we make it configurable?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] mkozioro commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
mkozioro commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-664168926
Hi @rvashishth
I'm also using AKS. 1.16.x and 1.17.x.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] toneill818 commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
toneill818 commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-672891558
@rvashishth The bug is in the labelSelector for podAntiAffinity.
```yaml
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- "{{ template "pulsar.name" . }}-{{ .Values.zookeeper.component }}" # This is where the issue is, it will never match
```
The app is set to pulsar.name-component. So it looks for app: pulsar-zookeeper, but the template for the labels sets the app to pulsar.name. Removing the "-component" should fix this. I can open a PR with this fix if you would like. Also with this change you will need to have as many nodes as the highest replicas for any of the stateful sets. Should we set the affinity to `preferredDuringSchedulingIgnoredDuringExecution`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] sijie commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-657651061
@mkozioro Thank you for reporting this issue. Are you interested in contributing a bug fix?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] sijie commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
sijie commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-658537932
@rvashishth sounds like a good idea. +1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] rvashishth commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
rvashishth commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-664347513
@mkozioro if you have a fix handly, please feel free to contribute.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] rvashishth commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
rvashishth commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-658587903
Sure. I can take this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] mkozioro commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
mkozioro commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-657689581
Sure, I can prepare PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-helm-chart] mkozioro commented on issue #39: Bug in podAntiAffinity labels
Posted by GitBox <gi...@apache.org>.
mkozioro commented on issue #39:
URL: https://github.com/apache/pulsar-helm-chart/issues/39#issuecomment-658585628
Hi @rvashishth
I suppose you have better knowledge about migrating to new helm 3 than me. I know only about changed labels. Maybe you would like to migrate this chart and with labels, fix podAntiAffinity?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org