You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by "Rashwan, Abderahman" <ab...@bell.ca> on 2018/04/04 20:51:50 UTC
zookeeper node can't join the cluster
Hello,
I have 2 servers, I installed proxmox in both and created a cluster contains 6 kafka nodes and 3 zookeepers
Server1: kafka1, kafka2, kafka3, zk1
Server2: kafka4, kafka5, kafka6, zk2
VM: zk3
When i shut down one server, for example server1 (kafka1, kafka2, kafka3, zk1) and then power it up Zk01 gives me an error and can't join the cluster, and I got this error
[2018-04-03 10:22:04,370] WARN Cannot open channel to 1 at election address zk001/172.31.254.56:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:562)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:479)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:379)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:757)
[2018-04-03 10:22:04,370] INFO Resolved hostname: zk001 to address: zk001/172.31.254.56 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2018-04-03 10:22:17,171] INFO Received connection request /172.31.254.56:58322 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2018-04-03 10:22:17,172] WARN Cannot open channel to 1 at election address zk001/172.31.254.56:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused (Connection refused)
When I restart the zookeeper service, it joined the cluster
Also when I start the zookeeper service after the boot with 10 sec, it worked
What could be the cause
Abderahman Rashwan
[bell]Bell Network | SOC
Network Security Engineering|Cyber Security Analyst
T: (514) 870-7001 M: (514) 443-5820
C: abderahman.rashwan@bell.ca<ma...@bell.ca>
RE: zookeeper node can't join the cluster
Posted by "Rashwan, Abderahman" <ab...@bell.ca>.
Hi,
Could it be something to do with Proxmox containers?
---------------
Could be but I tried VMs as well and gave me the same error
Which ZooKeeper version are u running?
Zookeeper version: 3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
Looks like you restarted zk01 and it was trying to connect to itself.
(zk001/172.31.254.56:3888)
Would you please attach your Zk config files too?
--------------
dataDir=/var/lib/zookeeper
clientPort=2181
maxClientCnxns=0
tickTime=2000
initLimit=5
syncLimit=2
server.1=zk001:2888:3888
server.2=zk002:2888:3888
server.3=zk003:2888:3888
------------
/etc/hosts file
172.31.254.57 zk002
172.31.254.56 zk001
172.31.254.10 zk003
Regards,
Andor
On Wed, Apr 4, 2018 at 10:51 PM, Rashwan, Abderahman < abderahman.rashwan@bell.ca> wrote:
> Hello,
>
>
>
> I have 2 servers, I installed proxmox in both and created a cluster
> contains 6 kafka nodes and 3 zookeepers
>
> Server1: kafka1, kafka2, kafka3, zk1
>
> Server2: kafka4, kafka5, kafka6, zk2
>
> VM: zk3
>
>
>
> When i shut down one server, for example server1 (kafka1, kafka2,
> kafka3, zk1) and then power it up Zk01 gives me an error and can’t join
> the cluster, and I got this error
>
>
>
> [2018-04-03 10:22:04,370] WARN Cannot open channel to 1 at election
> address zk001/172.31.254.56:3888 (org.apache.zookeeper.server.
> quorum.QuorumCnxManager)
>
> java.net.ConnectException: Connection refused (Connection refused)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:350)
>
> at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
>
> at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectOne(QuorumCnxManager.java:562)
>
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> handleConnection(QuorumCnxManager.java:479)
>
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> receiveConnection(QuorumCnxManager.java:379)
>
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> Listener.run(QuorumCnxManager.java:757)
>
> [2018-04-03 10:22:04,370] INFO Resolved hostname: zk001 to address:
> zk001/
> 172.31.254.56 (org.apache.zookeeper.server.quorum.QuorumPeer)
>
> [2018-04-03 10:22:17,171] INFO Received connection request /
> 172.31.254.56:58322
> (org.apache.zookeeper.server.quorum.QuorumCnxManager)
>
> [2018-04-03 10:22:17,172] WARN Cannot open channel to 1 at election
> address zk001/172.31.254.56:3888 (org.apache.zookeeper.server.
> quorum.QuorumCnxManager)
>
> java.net.ConnectException: Connection refused (Connection refused)
>
>
>
>
>
>
>
> When I restart the zookeeper service, it joined the cluster
>
>
>
> Also when I start the zookeeper service after the boot with 10 sec, it
> worked
>
>
>
> What could be the cause
>
>
>
> Abderahman Rashwan
>
> [image: bell]Bell Network | SOC
>
> Network Security Engineering|Cyber Security Analyst
>
> T: (514) 870-7001 M: (514) 443-5820
>
> C: abderahman.rashwan@bell.ca
>
>
>
Re: zookeeper node can't join the cluster
Posted by Andor Molnar <an...@cloudera.com>.
Hi,
Could it be something to do with Proxmox containers?
Which ZooKeeper version are u running?
Looks like you restarted zk01 and it was trying to connect to itself.
(zk001/172.31.254.56:3888)
Would you please attach your Zk config files too?
Regards,
Andor
On Wed, Apr 4, 2018 at 10:51 PM, Rashwan, Abderahman <
abderahman.rashwan@bell.ca> wrote:
> Hello,
>
>
>
> I have 2 servers, I installed proxmox in both and created a cluster
> contains 6 kafka nodes and 3 zookeepers
>
> Server1: kafka1, kafka2, kafka3, zk1
>
> Server2: kafka4, kafka5, kafka6, zk2
>
> VM: zk3
>
>
>
> When i shut down one server, for example server1 (kafka1, kafka2,
> kafka3, zk1) and then power it up Zk01 gives me an error and can’t join
> the cluster, and I got this error
>
>
>
> [2018-04-03 10:22:04,370] WARN Cannot open channel to 1 at election
> address zk001/172.31.254.56:3888 (org.apache.zookeeper.server.
> quorum.QuorumCnxManager)
>
> java.net.ConnectException: Connection refused (Connection refused)
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
>
> at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:350)
>
> at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
>
> at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:188)
>
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>
> at java.net.Socket.connect(Socket.java:589)
>
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectOne(QuorumCnxManager.java:562)
>
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> handleConnection(QuorumCnxManager.java:479)
>
> at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> receiveConnection(QuorumCnxManager.java:379)
>
> at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> Listener.run(QuorumCnxManager.java:757)
>
> [2018-04-03 10:22:04,370] INFO Resolved hostname: zk001 to address: zk001/
> 172.31.254.56 (org.apache.zookeeper.server.quorum.QuorumPeer)
>
> [2018-04-03 10:22:17,171] INFO Received connection request /
> 172.31.254.56:58322 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
>
> [2018-04-03 10:22:17,172] WARN Cannot open channel to 1 at election
> address zk001/172.31.254.56:3888 (org.apache.zookeeper.server.
> quorum.QuorumCnxManager)
>
> java.net.ConnectException: Connection refused (Connection refused)
>
>
>
>
>
>
>
> When I restart the zookeeper service, it joined the cluster
>
>
>
> Also when I start the zookeeper service after the boot with 10 sec, it
> worked
>
>
>
> What could be the cause
>
>
>
> Abderahman Rashwan
>
> [image: bell]Bell Network | SOC
>
> Network Security Engineering|Cyber Security Analyst
>
> T: (514) 870-7001 M: (514) 443-5820
>
> C: abderahman.rashwan@bell.ca
>
>
>