suport dynamic enable/disable health check #2947

[lordcheng10 <15...@qq.com.INVALID>]

Hi all,Recently I found a problem when upgrading pulsar and bookkeeper：
  When upgrading bookkeeper, need to restart the bookie nodes in turn. When restarting a bookie, the pulsar will fail to read and write to the bookie, so as to isolate the bookie node. The default isolation is 30 minutes. With more and more restarted bookie nodes, the traffic of the whole cluster will be transferred to the remaining few bookie nodes, At this time, the bookie node may be abnormal due to excessive pressure, such as pulsar read-write timeout.
  Therefore, I provide a configuration to dynamically turn off the health check. When upgrading bookkeeper, turn off the isolation function and turn it on after the upgrade, so as to avoid the impact of upgrading bookkeeper on the stability of pulsar.


I mentioned a PR：https://github.com/apache/bookkeeper/pull/2947

回复： suport dynamic enable/disable health check #2947

[lordcheng10 <15...@qq.com.INVALID>]

I don’t know if I have described it clearly, for example:
Here is a bookkeeper cluster with 3 nodes (b1, b2, b3), and the corresponding pulsar broker has 3 nodes (B1, B2, B3). When we restart the three nodes b1, b2, and b3, the pulsar broker will Put b1, b2, and b3 into the excludeBookies collection to isolate for 30 minutes. At this time, when the pulsar broker chooses the bookie node, it can only choose the b4 node, so the traffic of the entire cluster will go to the b4 node at this time





------------------ 原始邮件 ------------------
发件人: "Enrico Olivelli"<eolivelli@gmail.com&gt;; 
发送时间: 2021年12月21日(星期二) 晚上7:42
收件人: "dev"<dev@bookkeeper.apache.org&gt;; 
主题: Re: suport dynamic enable/disable health check #2947



Hello,
When you restart a bookie and the bookie announces itself as writable on
ZooKeeper, the clients should re-add it to the set of available bookies.

How many bookies do you have ?

Enrico

回复： suport dynamic enable/disable health check #2947

[lordcheng10 <15...@qq.com.INVALID>]

I don’t know if I have described it clearly, for example:
Here is a bookkeeper cluster with 3 nodes (b1, b2, b3), and the corresponding pulsar broker has 3 nodes (B1, B2, B3). When we restart the b1 and b2 nodes, the pulsar broker will switch b1, b2 Put it in the excludeBookies collection and isolate it for 30 minutes. At this time, when the pulsar broker chooses the bookie node, it can only choose the b3 node, so the traffic of the entire cluster will go to the b3 node at this time.

回复： suport dynamic enable/disable health check #2947

[lordcheng10 <15...@qq.com.INVALID>]

91 bookie nodes.
When a bookie node restarts, it will cause the pulsar broker to put it into the excludeBookies collection for isolation for 30 minutes. I reproduced it in the test cluster:







------------------ 原始邮件 ------------------
发件人: "Enrico Olivelli"<eolivelli@gmail.com&gt;; 
发送时间: 2021年12月21日(星期二) 晚上7:42
收件人: "dev"<dev@bookkeeper.apache.org&gt;; 
主题: Re: suport dynamic enable/disable health check #2947



Hello,
When you restart a bookie and the bookie announces itself as writable on
ZooKeeper, the clients should re-add it to the set of available bookies.

How many bookies do you have ?

Enrico

回复： suport dynamic enable/disable health check #2947

[lordcheng10 <15...@qq.com.INVALID>]

I don’t know if I have described it clearly, for example:
Here is a bookkeeper cluster with 3 nodes (b1, b2, b3), and the corresponding pulsar broker has 3 nodes (B1, B2, B3). When we restart the b1 and b2 nodes, the pulsar broker will switch b1, b2 Put it in the excludeBookies collection and isolate it for 30 minutes. At this time, when the pulsar broker chooses the bookie node, it can only choose the b3 node, so the traffic of the entire cluster will go to the b3 node at this time.

Re: suport dynamic enable/disable health check #2947

[Enrico Olivelli <eo...@gmail.com>]

Hello,
When you restart a bookie and the bookie announces itself as writable on
ZooKeeper, the clients should re-add it to the set of available bookies.

How many bookies do you have ?

Enrico