kafka/zookeepeer disconnection/reconnection

["Denis SALGON (contractor)" <De...@amadeus.com>]

CONFIDENTIAL & RESTRICTED

Hi Everybody,

I'm facing a problem I don't manage to find an explanation.
I've got ian openshift pod with kafka 0.10.2.1 and a zookeeper 3.4.9. We have only one topic feeded by  a kafka connector running in another pod that sends around 21 messages per second.
For some reasons, for the moment, the topic has only one partition and there's only one broker. The 3 specific kafka connector topics has been well created except for the number of replication because of the number of broker.

In kafka logs we have often (around 15 times a day) this kind incident (kafka seems to loose hos connection to zookeeper):
[ 2018-10-02 06:56:03,404] WARN Client session timed out, have not heard from server in 4036ms for sessionid 0x165fc77d1ae002e (org.apache.zookeeper.ClientCnxn)
[2018-10-02 06:56:03,431] INFO Client session timed out, have not heard from server in 4036ms for sessionid 0x165fc77d1ae002e, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-10-02 06:56:05,555] INFO Session establishment complete on server zookeeper-inval-cluster.default.svc.cluster.local/10.224.32.63:2181, sessionid = 0x165fc77d1ae002e, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2018-10-02 09:44:58,831] WARN Client session timed out, have not heard from server in 7070ms for sessionid 0x165fc77d1ae002e (org.apache.zookeeper.ClientCnxn)
[2018-10-02 09:44:58,832] INFO Client session timed out, have not heard from server in 7070ms for sessionid 0x165fc77d1ae002e, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-10-02 09:45:00,277] WARN Unable to reconnect to ZooKeeper service, session 0x165fc77d1ae002e has expired (org.apache.zookeeper.ClientCnxn)
[2018-10-02 09:45:00,277] INFO Unable to reconnect to ZooKeeper service, session 0x165fc77d1ae002e has expired, closing socket connection (org.apache.zookeeper.ClientCnxn)
[2018-10-02 09:45:01,844] INFO EventThread shut down for session: 0x165fc77d1ae002e (org.apache.zookeeper.ClientCnxn)

On the zookeeper side, we have this logs:
2018-10-02 06:56:03,487 [myid:] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x165fc77d1ae002e, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
        at java.lang.Thread.run(Thread.java:748)
2018-10-02 06:56:03,515 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket connection for client /10.225.11.1:37874 which had sessionid 0x165fc77d1ae002e
2018-10-02 06:56:05,518 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /10.225.11.1:34696
2018-10-02 06:56:05,545 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@921] - Client attempting to renew session 0x165fc77d1ae002e at /10.225.11.1:34696
2018-10-02 06:56:05,545 [myid:] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@673] - Established session 0x165fc77d1ae002e with negotiated timeout 6000 for client /10.225.11.1:34696

Ok, the reconnection works fine and externally, for our application, everything is ok.
But, I wondering if it could be the sign of something under sized or something that's going wrong. I 'googled' the problem of course, but I didn't find any satisfying explanation or solution.
Is there a anybody who faced this kind of behaviour of for giving something to investigate further ?

Anyway, thank you very much for any help or suggestion.

Denis.