Re: ZK servers without connections

[Brian Nixon <br...@gmail.com>]

Hi Gaël,

This might not necessarily be smelly though it depends on when the clients
join relative to an ensemble restart and when the leader is restarted.

Consider the following scenario with 1280 clients connected to a 5 node
ensemble. The number of connections to each server will be represented with
a 5-tuple.

In steady state, all clients come in randomly and the connections are
balanced (256, 256, 256, 256, 256).
The first server restarts and dumps its connections to the other four (0,
320, 320, 320, 320).
Same for the second (80, 0, 400, 400, 400).
Same for the third (180, 100, 0, 500, 500).
The fourth server happens to be a leader. All servers shed their
connections in a quick leader election and the clients rebalance (320, 320,
320, 0, 320).
The fifth server restarts and dumps its connections (400, 400, 400, 80, 0).

This is not too far from your reported allocations (300, 250, 300, 20, 1)
and the differences could be in noise or other restarts. Since ZooKeeper
does not automatically rebalance its client connections, all it would take
to create a lopsided connection count such as yours would be the restart of
two servers (the full ensemble restart is just one scenario that matches
the pattern). If your server lifespans are less than your client lifespans
then lopsided patterns like this will emerge. On the other hand, if your
clients are mostly short lived relative to your server lifespan then such a
pattern is an indicator that something is wrong.

-Brian


On Wed, Dec 18, 2019 at 5:23 AM Gael Jourdan-Weil <
gael.jourdan-weil@kelkoogroup.com> wrote:

> Hello,
>
> I'm using ZooKeeper in the context of SolrCloud servers and I'm wondering
> is some strange behaviors we have in Solr could be explained by issues in
> our ZooKeeper cluster.
>
> We have a 5 servers ZK cluster, but I noticed that:
> - 3 of those servers have between 250 and 300 alive connections
> - one other server have only 20 alive connections
> - and the last server only one alive connection
>
> Is it a normal behavior or is it smelly?
>
> Note that I checked the configurations and all clients should be using the
> 5 servers connection string.
>
> Thank you for any response,
>
> Gaël