You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Nick Vladiceanu <vl...@gmail.com> on 2022/10/17 09:14:59 UTC
Zookeeper >= 3.7 healtcheck random failures
hi all,
we’ve upgraded our Zookeeper that runs in Kubernetes (using bitnami helm chart) from version 3.6.1 to version 3.7.1 (also tried 3.8.0) and we’re observing random Liveness and Readiness failures:
Warning Unhealthy 100s (x2 over 5m10s) kubelet Liveness probe failed:
Tried with plain Zookeeper official image, same behaviour starting from the version >= 3.7.0.
Readiness and liveness probes are running the following script: exec [/bin/bash -c echo "ruok" | timeout 2 nc -w 2 localhost 2181 | grep imok]
Kubernetes version: 1.21.14
Couldn’t find anything in the ZK logs (not trace/debug mode though).
Did anyone else experience such issues when upgrading? We’ve returned back to the 3.6.1 and no failures are seen.
Thanks
Re: Zookeeper >= 3.7 healtcheck random failures
Posted by tison <wa...@gmail.com>.
Good to know.
Is it an `nc` shortcoming or we can flush the inflight response in some way
to ensure that the client sees the output?
Best,
tison.
Nick Vladiceanu <vl...@gmail.com> 于2022年10月18日周二 20:25写道：
> ok, switching away from “nc” seems to be helping, thank you.
>
> that was a bit of unexpected component to look at without network
> debugging..
>
> > On 18. Oct 2022, at 1:13 PM, Eugene Klimov <bl...@gmail.com>
> wrote:
> >
> > Replace nc to socat
> > or use pure bash
> >
> > bug on nc side
> > https://github.com/pravega/zookeeper-operator/pull/476
> >
> https://github.com/Altinity/clickhouse-operator/blob/0.20.0/deploy/zookeeper/quick-start-persistent-volume/zookeeper-1-node-for-test-probes.yaml#L188-L203
> >
> > пн, 17 окт. 2022 г. в 12:16, Nick Vladiceanu <vl...@gmail.com>:
> >>
> >> hi all,
> >> we’ve upgraded our Zookeeper that runs in Kubernetes (using bitnami
> helm chart) from version 3.6.1 to version 3.7.1 (also tried 3.8.0) and
> we’re observing random Liveness and Readiness failures:
> >>
> >> Warning Unhealthy 100s (x2 over 5m10s) kubelet Liveness
> probe failed:
> >>
> >> Tried with plain Zookeeper official image, same behaviour starting from
> the version >= 3.7.0.
> >>
> >> Readiness and liveness probes are running the following script: exec
> [/bin/bash -c echo "ruok" | timeout 2 nc -w 2 localhost 2181 | grep imok]
> >> Kubernetes version: 1.21.14
> >>
> >> Couldn’t find anything in the ZK logs (not trace/debug mode though).
> >>
> >> Did anyone else experience such issues when upgrading? We’ve returned
> back to the 3.6.1 and no failures are seen.
> >>
> >> Thanks
>
>
Re: Zookeeper >= 3.7 healtcheck random failures
Posted by Nick Vladiceanu <vl...@gmail.com>.
ok, switching away from “nc” seems to be helping, thank you.
that was a bit of unexpected component to look at without network debugging..
> On 18. Oct 2022, at 1:13 PM, Eugene Klimov <bl...@gmail.com> wrote:
>
> Replace nc to socat
> or use pure bash
>
> bug on nc side
> https://github.com/pravega/zookeeper-operator/pull/476
> https://github.com/Altinity/clickhouse-operator/blob/0.20.0/deploy/zookeeper/quick-start-persistent-volume/zookeeper-1-node-for-test-probes.yaml#L188-L203
>
> пн, 17 окт. 2022 г. в 12:16, Nick Vladiceanu <vl...@gmail.com>:
>>
>> hi all,
>> we’ve upgraded our Zookeeper that runs in Kubernetes (using bitnami helm chart) from version 3.6.1 to version 3.7.1 (also tried 3.8.0) and we’re observing random Liveness and Readiness failures:
>>
>> Warning Unhealthy 100s (x2 over 5m10s) kubelet Liveness probe failed:
>>
>> Tried with plain Zookeeper official image, same behaviour starting from the version >= 3.7.0.
>>
>> Readiness and liveness probes are running the following script: exec [/bin/bash -c echo "ruok" | timeout 2 nc -w 2 localhost 2181 | grep imok]
>> Kubernetes version: 1.21.14
>>
>> Couldn’t find anything in the ZK logs (not trace/debug mode though).
>>
>> Did anyone else experience such issues when upgrading? We’ve returned back to the 3.6.1 and no failures are seen.
>>
>> Thanks
Re: Zookeeper >= 3.7 healtcheck random failures
Posted by Eugene Klimov <bl...@gmail.com>.
Replace nc to socat
or use pure bash
bug on nc side
https://github.com/pravega/zookeeper-operator/pull/476
https://github.com/Altinity/clickhouse-operator/blob/0.20.0/deploy/zookeeper/quick-start-persistent-volume/zookeeper-1-node-for-test-probes.yaml#L188-L203
пн, 17 окт. 2022 г. в 12:16, Nick Vladiceanu <vl...@gmail.com>:
>
> hi all,
> we’ve upgraded our Zookeeper that runs in Kubernetes (using bitnami helm chart) from version 3.6.1 to version 3.7.1 (also tried 3.8.0) and we’re observing random Liveness and Readiness failures:
>
> Warning Unhealthy 100s (x2 over 5m10s) kubelet Liveness probe failed:
>
> Tried with plain Zookeeper official image, same behaviour starting from the version >= 3.7.0.
>
> Readiness and liveness probes are running the following script: exec [/bin/bash -c echo "ruok" | timeout 2 nc -w 2 localhost 2181 | grep imok]
> Kubernetes version: 1.21.14
>
> Couldn’t find anything in the ZK logs (not trace/debug mode though).
>
> Did anyone else experience such issues when upgrading? We’ve returned back to the 3.6.1 and no failures are seen.
>
> Thanks