Issue of one kafka broker not listed in zookeeper

[Maharajan S <ma...@gmail.com>]

Hi,

We have incident that one of the 10 brokers not listed in brokers list of
zookeeper.

This is verified by running following command

>> echo dump | nc cz2 2181

SessionTracker dump:

Session Sets (4):

0 expire at Fri Jan 13 22:32:14 EST 2017:

0 expire at Fri Jan 13 22:32:16 EST 2017:

7 expire at Fri Jan 13 22:32:18 EST 2017:

        0x259968e41e30000

        0x35996670d5d0001

        0x35996670d5d0000

        0x159966708470004

        0x159966e47760000

        0x159966708470003

        0x2599672df260000

3 expire at Fri Jan 13 22:32:20 EST 2017:

        0x159968e41dd0000

        0x259966708550001

        0x259966708550000

ephemeral nodes dump:

Sessions with Ephemerals (9):

0x259966708550000:

        /brokers/ids/112

0x259968e41e30000:

        /brokers/ids/213

0x159968e41dd0000:

        /brokers/ids/19

0x159966708470003:

        /brokers/ids/110

0x35996670d5d0000:

        /brokers/ids/113

        /controller

0x259966708550001:

        /brokers/ids/111

0x159966708470004:

        /brokers/ids/212

0x2599672df260000:

        /brokers/ids/29

0x35996670d5d0001:

        /brokers/ids/210

------

There are 10 sessions, but only 9 sessions are listed with brokers.
Broker with id 211 is not listed. Session 0x159966e47760000 is not shown
with broker id 211.

In the broker side log, I do see it is connected

>> zgrep "0x159966e47760000" *log*



zk.log:[2017-01-13 01:05:28,513] INFO Session establishment complete on
server cz1/169.254.2.19:2181, sessionid = 0x159966e47760000, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:35:38,163] INFO Unable to read additional data from
server sessionid 0x159966e47760000, likely server has closed socket,
closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:35:39,101] WARN Session 0x159966e47760000 for server
null, unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:35:40,121] INFO Unable to read additional data from
server sessionid 0x159966e47760000, likely server has closed socket,
closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:35:41,770] WARN Session 0x159966e47760000 for server
null, unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:35:42,439] WARN Session 0x159966e47760000 for server
null, unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:35:43,235] INFO Unable to read additional data from
server sessionid 0x159966e47760000, likely server has closed socket,
closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:35:44,950] WARN Session 0x159966e47760000 for server
null, unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:35:45,837] INFO Unable to read additional data from
server sessionid 0x159966e47760000, likely server has closed socket,
closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

.

.

.

.

zk.log:[2017-01-13 01:40:14,818] INFO Unable to read additional data from
server sessionid 0x159966e47760000, likely server has closed socket,
closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:40:15,916] WARN Session 0x159966e47760000 for server
null, unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:40:19,692] INFO Client session timed out, have not
heard from server in 3676ms for sessionid 0x159966e47760000, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:40:20,632] INFO Unable to read additional data from
server sessionid 0x159966e47760000, likely server has closed socket,
closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:40:20,814] WARN Session 0x159966e47760000 for server
null, unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:40:22,089] WARN Session 0x159966e47760000 for server
null, unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)

zk.log:[2017-01-13 01:40:22,562] INFO Session establishment complete on
server cz2/169.254.2.29:2181, sessionid = 0x159966e47760000, negotiated
timeout = 6000 (org.apache.zookeeper.ClientCnxn)

After several tries, broker connected with zookeeper cz2:2181. I am not
sure how to debug this issue. It would be helpful if someone provides way
to debug this issue.

Regards,
Maharajan S