You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by penguin <ba...@wellsfargo.com> on 2015/03/11 16:02:23 UTC
RE: New Zookeeper Installation - connection refused
Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK instances setup on the same physical machine, so no need to worry about whether multiple physical machines can communicate from one to the other, right?
My setup on the Linux sever is as follows:
Directory for instance 1: /home/zk1
Directory for instance 2: /home/zk2
Directory for instance 3: /home/zk3
In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2, zk3 for the other instances):
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/zk1/zookeeper-3.4.6/data
dataLogDir=/home/zk1/zookeeper-3.4.6/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.6=localhost:2888:3888
server.7=localhost:2889:3889
server.8=localhost:2890:3890
the myid files are 6, 7 and 8 respectively for each instance. Is there something I’m missing because I’m still getting the connection refused warning, and the status of the zk servers doesn’t show anything but an error.
Zookeeper.out shows:
2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=6]:QuorumCnxManager@38
2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
pl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
82)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
zkServer.sh status shows:
$ ./zkServer.sh status
JMX enabled by default
Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
Error contacting service. It is probably not running.
$ ./zkServer.sh start
JMX enabled by default
Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 27102.
Regards,
Barry
From: Flavio Junqueira-2 [via zookeeper-user] [mailto:ml-node+s578899n7580878h66@n2.nabble.com]
Sent: Tuesday, February 24, 2015 6:53 PM
To: Barnett, Barry F
Subject: Re: New Zookeeper Installation - connection refused
It's hard to say what the problem is without more information. For leader election, the servers will try to create a TCP connection with each other, so if you have tested separately and your sample code worked for the same ports, the only thing I can think of is that there is a problem with the server name resolution. Is it possible that the ZK servers aren't resolving the names of your servers appropriately?
Perhaps you could also use tcpdump or some similar tool to inspect the network traffic. In fact, I think someone here has written a tool to inspect the leader election traffic and display it. Was it Raul?
-Flavio
> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>> wrote:
>
> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither work, both receiving the ‘connection refused’ error.
>
> I’m now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>
> Myid file is set to 1,2 and 3 respectively on the machines.
> Zoo.cfg has the following, and uses 2181 as its client port:
>
> server.1=servername:2888:3888
> server.2=servername:2888:3888
> server.3=servername:2888:3888
>
> Interestingly enough, when I start up the server1 instance, the zookeeper.out file receives ongoing information on the connection refused to each server instance. When I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information. Is that normal?
>
> In any case, I’ve checked the ports by using nc –v hostname port, and connections are successful. So this is not a firewall issue or port block issue. I’ve had my OSE’s check the servers as well to ensure no communications issue.
>
> *Note, port 2181 is an eForward port. Is this supposed to be used by Zookeeper?
>
> Same errors over and over:
> 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING
> 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815] - New e
> lection. My id = 1, proposed zxid=0x0
> 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification:
> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
> (n.peerEpoch) LOOKING (my state)
> 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channe
> l to 2 at election address servername:3888
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>
> Regards,
>
> Barry
>
> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1> <mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2>>]
> Sent: Wednesday, February 04, 2015 4:17 PM
> To: Barnett, Barry F
> Subject: Re: New Zookeeper Installation - connection refused
>
> That's because some client is trying to connect to the server, but the server is looking (not following or leading) and the result of the nc run confirms it. I think you still have a connectivity problem. There is something preventing the servers from connecting to each other and it seems to be something related to your environment.
>
> -Flavio
>
>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>> wrote:
>>
>> I now made this a 3 server ensemble as opposed to a 5 server ensemble. Now getting different messages in my out logs:P
>>
>> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>
>> $ ./zkServer.sh status
>> JMX enabled by default
>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>> Error contacting service. It is probably not running.
>>
>>
>> $ echo mntr | nc localhost 2181
>> This ZooKeeper instance is not currently serving requests
>>
>>
>>
>> Regards,
>>
>> Barry
>>
>>
>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>> Sent: Wednesday, February 04, 2015 10:56 AM
>> To: Barnett, Barry F
>> Subject: Re: New Zookeeper Installation - connection refused
>>
>> Barry,
>> looks like you want to go to the machine where Zk is running and check
>> with "netstat -an" or some other tool on which port Zk is really
>> running... unless this is a NAT issue ;-)
>>
>> Cheers,
>> --Jürgen
>>
>> On 04.02.2015 16:51, penguin wrote:
>>> $ nc -v remotehostname 2181
>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection refused
>>>
>>> Regards,
>>>
>>> Barry
>>>
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>> --
>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
> To unsubscribe from New Zookeeper Installation - connection refused, click here< href="" target="_top" rel="nofollow" link="external">
> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml <http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>
>
>
>
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
________________________________
If you reply to this email, your message will be added to the discussion below:
http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
RE: New Zookeeper Installation - connection refused
Posted by penguin <ba...@wellsfargo.com>.
If I use the same zoo.cfg file that I use for the instances running on the same physical machine, for the remote physical machines zk servers, they fail with connection refused. All these machines are in the same network, no firewalls between, no ports blocked. Very confusing.
Regards,
Barry
From: Michi Mutsuzaki [via zookeeper-user] [mailto:ml-node+s578899n7580914h82@n2.nabble.com]
Sent: Friday, March 13, 2015 2:56 AM
To: Barnett, Barry F
Subject: Re: New Zookeeper Installation - connection refused
Hi Barry,
You don't need to use different client ports if you are running
ZooKeeper servers on separate machines.
On Thu, Mar 12, 2015 at 4:45 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580914&i=0>> wrote:
> Ok, the client port change worked for my connection refused issue with instances on the same server. I now have 2 followers and 1 leader. So I've determined that zk can run 3 instances on the same physical machine with no connection issues. I was getting the same connection refused error when I had 3 separate machines running 1 instance of zk on each, trying to instantiate an ensemble. Should I still have different client ports in that scenario? I had the same client ports and it was failing miserably. I am not familiar with zkconf, but I will look at it.
>
> Regards,
>
> Barry
>
> From: Michi Mutsuzaki [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580914&i=1>]
> Sent: Thursday, March 12, 2015 1:26 AM
> To: Barnett, Barry F
> Subject: Re: New Zookeeper Installation - connection refused
>
> Yes that's even better :)
>
> On Wed, Mar 11, 2015 at 10:22 PM, Patrick Hunt <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=0>> wrote:
>
>> Consider zkconf? https://github.com/phunt/zkconf
>>
>> Patrick
>>
>> On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=1>> wrote:
>>> The clientPort parameter needs to be different for each instance if
>>> you are running them on a same box. Something like this should work:
>>>
>>> https://paste.apache.org/n59c?action=download
>>>
>>> On Wed, Mar 11, 2015 at 8:02 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=2>> wrote:
>>>> Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK instances setup on the same physical machine, so no need to worry about whether multiple physical machines can communicate from one to the other, right?
>>>> My setup on the Linux sever is as follows:
>>>>
>>>> Directory for instance 1: /home/zk1
>>>> Directory for instance 2: /home/zk2
>>>> Directory for instance 3: /home/zk3
>>>>
>>>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2, zk3 for the other instances):
>>>> # The number of milliseconds of each tick
>>>> tickTime=2000
>>>> # The number of ticks that the initial
>>>> # synchronization phase can take
>>>> initLimit=10
>>>> # The number of ticks that can pass between
>>>> # sending a request and getting an acknowledgement
>>>> syncLimit=5
>>>> # the directory where the snapshot is stored.
>>>> # do not use /tmp for storage, /tmp here is just
>>>> # example sakes.
>>>> dataDir=/home/zk1/zookeeper-3.4.6/data
>>>> dataLogDir=/home/zk1/zookeeper-3.4.6/data
>>>> # the port at which the clients will connect
>>>> clientPort=2181
>>>> # the maximum number of client connections.
>>>> maxClientCnxns=60
>>>> #
>>>> # Be sure to read the maintenance section of the
>>>> # administrator guide before turning on autopurge.
>>>> #
>>>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>> #
>>>> # The number of snapshots to retain in dataDir
>>>> #autopurge.snapRetainCount=3
>>>> # Purge task interval in hours
>>>> # Set to "0" to disable auto purge feature
>>>> #autopurge.purgeInterval=1
>>>> server.6=localhost:2888:3888
>>>> server.7=localhost:2889:3889
>>>> server.8=localhost:2890:3890
>>>>
>>>> the myid files are 6, 7 and 8 respectively for each instance. Is there something I'm missing because I'm still getting the connection refused warning, and the status of the zk servers doesn't show anything but an error.
>>>>
>>>> Zookeeper.out shows:
>>>> 2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=6]:QuorumCnxManager@38
>>>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>>>> :339)
>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>>>> pl.java:200)
>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>>>> 82)
>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>> at java.net.Socket.connect(Socket.java:579)
>>>>
>>>> zkServer.sh status shows:
>>>>
>>>> $ ./zkServer.sh status
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Error contacting service. It is probably not running.
>>>> $ ./zkServer.sh start
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Starting zookeeper ... already running as process 27102.
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=3>]
>>>> Sent: Tuesday, February 24, 2015 6:53 PM
>>>> To: Barnett, Barry F
>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>
>>>> It's hard to say what the problem is without more information. For leader election, the servers will try to create a TCP connection with each other, so if you have tested separately and your sample code worked for the same ports, the only thing I can think of is that there is a problem with the server name resolution. Is it possible that the ZK servers aren't resolving the names of your servers appropriately?
>>>>
>>>> Perhaps you could also use tcpdump or some similar tool to inspect the network traffic. In fact, I think someone here has written a tool to inspect the leader election traffic and display it. Was it Raul?
>>>>
>>>> -Flavio
>>>>
>>>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>> wrote:
>>>>>
>>>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither work, both receiving the 'connection refused' error.
>>>>>
>>>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>>>>
>>>>> Myid file is set to 1,2 and 3 respectively on the machines.
>>>>> Zoo.cfg has the following, and uses 2181 as its client port:
>>>>>
>>>>> server.1=servername:2888:3888
>>>>> server.2=servername:2888:3888
>>>>> server.3=servername:2888:3888
>>>>>
>>>>> Interestingly enough, when I start up the server1 instance, the zookeeper.out file receives ongoing information on the connection refused to each server instance. When I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information. Is that normal?
>>>>>
>>>>> In any case, I've checked the ports by using nc -v hostname port, and connections are successful. So this is not a firewall issue or port block issue. I've had my OSE's check the servers as well to ensure no communications issue.
>>>>>
>>>>> *Note, port 2181 is an eForward port. Is this supposed to be used by Zookeeper?
>>>>>
>>>>> Same errors over and over:
>>>>> 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING
>>>>> 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815] - New e
>>>>> lection. My id = 1, proposed zxid=0x0
>>>>> 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification:
>>>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
>>>>> (n.peerEpoch) LOOKING (my state)
>>>>> 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channe
>>>>> l to 2 at election address servername:3888
>>>>> java.net.ConnectException: Connection refused
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>>>
>>>>> Regards,
>>>>>
>>>>> Barry
>>>>>
>>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1> <mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2<mailto:[hidden%20email]%3c/user/SendEmail.jtp?type=node&node=7580878&i=2>>>]
>>>>> Sent: Wednesday, February 04, 2015 4:17 PM
>>>>> To: Barnett, Barry F
>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>
>>>>> That's because some client is trying to connect to the server, but the server is looking (not following or leading) and the result of the nc run confirms it. I think you still have a connectivity problem. There is something preventing the servers from connecting to each other and it seems to be something related to your environment.
>>>>>
>>>>> -Flavio
>>>>>
>>>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>> wrote:
>>>>>>
>>>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble. Now getting different messages in my out logs:P
>>>>>>
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>>>>
>>>>>> $ ./zkServer.sh status
>>>>>> JMX enabled by default
>>>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>>>>> Error contacting service. It is probably not running.
>>>>>>
>>>>>>
>>>>>> $ echo mntr | nc localhost 2181
>>>>>> This ZooKeeper instance is not currently serving requests
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>>>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>>>>> To: Barnett, Barry F
>>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>>
>>>>>> Barry,
>>>>>> looks like you want to go to the machine where Zk is running and check
>>>>>> with "netstat -an" or some other tool on which port Zk is really
>>>>>> running... unless this is a NAT issue ;-)
>>>>>>
>>>>>> Cheers,
>>>>>> --Jürgen
>>>>>>
>>>>>> On 04.02.2015 16:51, penguin wrote:
>>>>>>> $ nc -v remotehostname 2181
>>>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection refused
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> If you reply to this email, your message will be added to the discussion below:
>>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ________________________________
>>>>> If you reply to this email, your message will be added to the discussion below:
>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here< href="" target="_top" rel="nofollow" link="external">
>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml <http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>>>>
>>>>
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion below:
>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580909.html
> To unsubscribe from New Zookeeper Installation - connection refused, click here<
> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580912.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
________________________________
If you reply to this email, your message will be added to the discussion below:
http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580914.html
To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580921.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
RE: New Zookeeper Installation - connection refused
Posted by penguin <ba...@wellsfargo.com>.
Ok, I tried 2888 for my client port, 3181 and 4181 for my other ports (Quorom, etc), and I just setup 2 instances, 1 on each physical machine. I get the ‘connection refused’ error again. Nothing is using any of the ports. When I do a netstat –a|grep 2888 on the first instance, it shows the port has a listen on it now that I started up that instance, which is good. When I do the same on the 2nd instance, I get nothing (odd right?). In any case, I get the connection refused with the 1rst instance trying to connect to the second on port 4181, and neither show up in my netstat command as being used.
Regards,
Barry
From: Michi Mutsuzaki [via zookeeper-user] [mailto:ml-node+s578899n7580914h82@n2.nabble.com]
Sent: Friday, March 13, 2015 2:56 AM
To: Barnett, Barry F
Subject: Re: New Zookeeper Installation - connection refused
Hi Barry,
You don't need to use different client ports if you are running
ZooKeeper servers on separate machines.
On Thu, Mar 12, 2015 at 4:45 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580914&i=0>> wrote:
> Ok, the client port change worked for my connection refused issue with instances on the same server. I now have 2 followers and 1 leader. So I've determined that zk can run 3 instances on the same physical machine with no connection issues. I was getting the same connection refused error when I had 3 separate machines running 1 instance of zk on each, trying to instantiate an ensemble. Should I still have different client ports in that scenario? I had the same client ports and it was failing miserably. I am not familiar with zkconf, but I will look at it.
>
> Regards,
>
> Barry
>
> From: Michi Mutsuzaki [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580914&i=1>]
> Sent: Thursday, March 12, 2015 1:26 AM
> To: Barnett, Barry F
> Subject: Re: New Zookeeper Installation - connection refused
>
> Yes that's even better :)
>
> On Wed, Mar 11, 2015 at 10:22 PM, Patrick Hunt <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=0>> wrote:
>
>> Consider zkconf? https://github.com/phunt/zkconf
>>
>> Patrick
>>
>> On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=1>> wrote:
>>> The clientPort parameter needs to be different for each instance if
>>> you are running them on a same box. Something like this should work:
>>>
>>> https://paste.apache.org/n59c?action=download
>>>
>>> On Wed, Mar 11, 2015 at 8:02 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=2>> wrote:
>>>> Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK instances setup on the same physical machine, so no need to worry about whether multiple physical machines can communicate from one to the other, right?
>>>> My setup on the Linux sever is as follows:
>>>>
>>>> Directory for instance 1: /home/zk1
>>>> Directory for instance 2: /home/zk2
>>>> Directory for instance 3: /home/zk3
>>>>
>>>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2, zk3 for the other instances):
>>>> # The number of milliseconds of each tick
>>>> tickTime=2000
>>>> # The number of ticks that the initial
>>>> # synchronization phase can take
>>>> initLimit=10
>>>> # The number of ticks that can pass between
>>>> # sending a request and getting an acknowledgement
>>>> syncLimit=5
>>>> # the directory where the snapshot is stored.
>>>> # do not use /tmp for storage, /tmp here is just
>>>> # example sakes.
>>>> dataDir=/home/zk1/zookeeper-3.4.6/data
>>>> dataLogDir=/home/zk1/zookeeper-3.4.6/data
>>>> # the port at which the clients will connect
>>>> clientPort=2181
>>>> # the maximum number of client connections.
>>>> maxClientCnxns=60
>>>> #
>>>> # Be sure to read the maintenance section of the
>>>> # administrator guide before turning on autopurge.
>>>> #
>>>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>> #
>>>> # The number of snapshots to retain in dataDir
>>>> #autopurge.snapRetainCount=3
>>>> # Purge task interval in hours
>>>> # Set to "0" to disable auto purge feature
>>>> #autopurge.purgeInterval=1
>>>> server.6=localhost:2888:3888
>>>> server.7=localhost:2889:3889
>>>> server.8=localhost:2890:3890
>>>>
>>>> the myid files are 6, 7 and 8 respectively for each instance. Is there something I'm missing because I'm still getting the connection refused warning, and the status of the zk servers doesn't show anything but an error.
>>>>
>>>> Zookeeper.out shows:
>>>> 2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=6]:QuorumCnxManager@38
>>>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>>>> :339)
>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>>>> pl.java:200)
>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>>>> 82)
>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>> at java.net.Socket.connect(Socket.java:579)
>>>>
>>>> zkServer.sh status shows:
>>>>
>>>> $ ./zkServer.sh status
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Error contacting service. It is probably not running.
>>>> $ ./zkServer.sh start
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Starting zookeeper ... already running as process 27102.
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=3>]
>>>> Sent: Tuesday, February 24, 2015 6:53 PM
>>>> To: Barnett, Barry F
>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>
>>>> It's hard to say what the problem is without more information. For leader election, the servers will try to create a TCP connection with each other, so if you have tested separately and your sample code worked for the same ports, the only thing I can think of is that there is a problem with the server name resolution. Is it possible that the ZK servers aren't resolving the names of your servers appropriately?
>>>>
>>>> Perhaps you could also use tcpdump or some similar tool to inspect the network traffic. In fact, I think someone here has written a tool to inspect the leader election traffic and display it. Was it Raul?
>>>>
>>>> -Flavio
>>>>
>>>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>> wrote:
>>>>>
>>>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither work, both receiving the 'connection refused' error.
>>>>>
>>>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>>>>
>>>>> Myid file is set to 1,2 and 3 respectively on the machines.
>>>>> Zoo.cfg has the following, and uses 2181 as its client port:
>>>>>
>>>>> server.1=servername:2888:3888
>>>>> server.2=servername:2888:3888
>>>>> server.3=servername:2888:3888
>>>>>
>>>>> Interestingly enough, when I start up the server1 instance, the zookeeper.out file receives ongoing information on the connection refused to each server instance. When I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information. Is that normal?
>>>>>
>>>>> In any case, I've checked the ports by using nc -v hostname port, and connections are successful. So this is not a firewall issue or port block issue. I've had my OSE's check the servers as well to ensure no communications issue.
>>>>>
>>>>> *Note, port 2181 is an eForward port. Is this supposed to be used by Zookeeper?
>>>>>
>>>>> Same errors over and over:
>>>>> 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING
>>>>> 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815] - New e
>>>>> lection. My id = 1, proposed zxid=0x0
>>>>> 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification:
>>>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
>>>>> (n.peerEpoch) LOOKING (my state)
>>>>> 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channe
>>>>> l to 2 at election address servername:3888
>>>>> java.net.ConnectException: Connection refused
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>>>
>>>>> Regards,
>>>>>
>>>>> Barry
>>>>>
>>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1> <mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2<mailto:[hidden%20email]%3c/user/SendEmail.jtp?type=node&node=7580878&i=2>>>]
>>>>> Sent: Wednesday, February 04, 2015 4:17 PM
>>>>> To: Barnett, Barry F
>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>
>>>>> That's because some client is trying to connect to the server, but the server is looking (not following or leading) and the result of the nc run confirms it. I think you still have a connectivity problem. There is something preventing the servers from connecting to each other and it seems to be something related to your environment.
>>>>>
>>>>> -Flavio
>>>>>
>>>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>> wrote:
>>>>>>
>>>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble. Now getting different messages in my out logs:P
>>>>>>
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>>>>
>>>>>> $ ./zkServer.sh status
>>>>>> JMX enabled by default
>>>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>>>>> Error contacting service. It is probably not running.
>>>>>>
>>>>>>
>>>>>> $ echo mntr | nc localhost 2181
>>>>>> This ZooKeeper instance is not currently serving requests
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>>>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>>>>> To: Barnett, Barry F
>>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>>
>>>>>> Barry,
>>>>>> looks like you want to go to the machine where Zk is running and check
>>>>>> with "netstat -an" or some other tool on which port Zk is really
>>>>>> running... unless this is a NAT issue ;-)
>>>>>>
>>>>>> Cheers,
>>>>>> --Jürgen
>>>>>>
>>>>>> On 04.02.2015 16:51, penguin wrote:
>>>>>>> $ nc -v remotehostname 2181
>>>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection refused
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> If you reply to this email, your message will be added to the discussion below:
>>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ________________________________
>>>>> If you reply to this email, your message will be added to the discussion below:
>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here< href="" target="_top" rel="nofollow" link="external">
>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml <http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>>>>
>>>>
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion below:
>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580909.html
> To unsubscribe from New Zookeeper Installation - connection refused, click here<
> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580912.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
________________________________
If you reply to this email, your message will be added to the discussion below:
http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580914.html
To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580915.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: New Zookeeper Installation - connection refused
Posted by Michi Mutsuzaki <mi...@cs.stanford.edu>.
Hi Barry,
You don't need to use different client ports if you are running
ZooKeeper servers on separate machines.
On Thu, Mar 12, 2015 at 4:45 AM, penguin <ba...@wellsfargo.com> wrote:
> Ok, the client port change worked for my connection refused issue with instances on the same server. I now have 2 followers and 1 leader. So I've determined that zk can run 3 instances on the same physical machine with no connection issues. I was getting the same connection refused error when I had 3 separate machines running 1 instance of zk on each, trying to instantiate an ensemble. Should I still have different client ports in that scenario? I had the same client ports and it was failing miserably. I am not familiar with zkconf, but I will look at it.
>
> Regards,
>
> Barry
>
> From: Michi Mutsuzaki [via zookeeper-user] [mailto:ml-node+s578899n7580909h27@n2.nabble.com]
> Sent: Thursday, March 12, 2015 1:26 AM
> To: Barnett, Barry F
> Subject: Re: New Zookeeper Installation - connection refused
>
> Yes that's even better :)
>
> On Wed, Mar 11, 2015 at 10:22 PM, Patrick Hunt <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=0>> wrote:
>
>> Consider zkconf? https://github.com/phunt/zkconf
>>
>> Patrick
>>
>> On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=1>> wrote:
>>> The clientPort parameter needs to be different for each instance if
>>> you are running them on a same box. Something like this should work:
>>>
>>> https://paste.apache.org/n59c?action=download
>>>
>>> On Wed, Mar 11, 2015 at 8:02 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=2>> wrote:
>>>> Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK instances setup on the same physical machine, so no need to worry about whether multiple physical machines can communicate from one to the other, right?
>>>> My setup on the Linux sever is as follows:
>>>>
>>>> Directory for instance 1: /home/zk1
>>>> Directory for instance 2: /home/zk2
>>>> Directory for instance 3: /home/zk3
>>>>
>>>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2, zk3 for the other instances):
>>>> # The number of milliseconds of each tick
>>>> tickTime=2000
>>>> # The number of ticks that the initial
>>>> # synchronization phase can take
>>>> initLimit=10
>>>> # The number of ticks that can pass between
>>>> # sending a request and getting an acknowledgement
>>>> syncLimit=5
>>>> # the directory where the snapshot is stored.
>>>> # do not use /tmp for storage, /tmp here is just
>>>> # example sakes.
>>>> dataDir=/home/zk1/zookeeper-3.4.6/data
>>>> dataLogDir=/home/zk1/zookeeper-3.4.6/data
>>>> # the port at which the clients will connect
>>>> clientPort=2181
>>>> # the maximum number of client connections.
>>>> maxClientCnxns=60
>>>> #
>>>> # Be sure to read the maintenance section of the
>>>> # administrator guide before turning on autopurge.
>>>> #
>>>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>> #
>>>> # The number of snapshots to retain in dataDir
>>>> #autopurge.snapRetainCount=3
>>>> # Purge task interval in hours
>>>> # Set to "0" to disable auto purge feature
>>>> #autopurge.purgeInterval=1
>>>> server.6=localhost:2888:3888
>>>> server.7=localhost:2889:3889
>>>> server.8=localhost:2890:3890
>>>>
>>>> the myid files are 6, 7 and 8 respectively for each instance. Is there something I'm missing because I'm still getting the connection refused warning, and the status of the zk servers doesn't show anything but an error.
>>>>
>>>> Zookeeper.out shows:
>>>> 2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=6]:QuorumCnxManager@38
>>>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>>>> :339)
>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>>>> pl.java:200)
>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>>>> 82)
>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>>> at java.net.Socket.connect(Socket.java:579)
>>>>
>>>> zkServer.sh status shows:
>>>>
>>>> $ ./zkServer.sh status
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Error contacting service. It is probably not running.
>>>> $ ./zkServer.sh start
>>>> JMX enabled by default
>>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>>> Starting zookeeper ... already running as process 27102.
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=3>]
>>>> Sent: Tuesday, February 24, 2015 6:53 PM
>>>> To: Barnett, Barry F
>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>
>>>> It's hard to say what the problem is without more information. For leader election, the servers will try to create a TCP connection with each other, so if you have tested separately and your sample code worked for the same ports, the only thing I can think of is that there is a problem with the server name resolution. Is it possible that the ZK servers aren't resolving the names of your servers appropriately?
>>>>
>>>> Perhaps you could also use tcpdump or some similar tool to inspect the network traffic. In fact, I think someone here has written a tool to inspect the leader election traffic and display it. Was it Raul?
>>>>
>>>> -Flavio
>>>>
>>>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>> wrote:
>>>>>
>>>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither work, both receiving the 'connection refused' error.
>>>>>
>>>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>>>>
>>>>> Myid file is set to 1,2 and 3 respectively on the machines.
>>>>> Zoo.cfg has the following, and uses 2181 as its client port:
>>>>>
>>>>> server.1=servername:2888:3888
>>>>> server.2=servername:2888:3888
>>>>> server.3=servername:2888:3888
>>>>>
>>>>> Interestingly enough, when I start up the server1 instance, the zookeeper.out file receives ongoing information on the connection refused to each server instance. When I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information. Is that normal?
>>>>>
>>>>> In any case, I've checked the ports by using nc -v hostname port, and connections are successful. So this is not a firewall issue or port block issue. I've had my OSE's check the servers as well to ensure no communications issue.
>>>>>
>>>>> *Note, port 2181 is an eForward port. Is this supposed to be used by Zookeeper?
>>>>>
>>>>> Same errors over and over:
>>>>> 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING
>>>>> 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815] - New e
>>>>> lection. My id = 1, proposed zxid=0x0
>>>>> 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification:
>>>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
>>>>> (n.peerEpoch) LOOKING (my state)
>>>>> 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channe
>>>>> l to 2 at election address servername:3888
>>>>> java.net.ConnectException: Connection refused
>>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>>>
>>>>> Regards,
>>>>>
>>>>> Barry
>>>>>
>>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1> <mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2>>]
>>>>> Sent: Wednesday, February 04, 2015 4:17 PM
>>>>> To: Barnett, Barry F
>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>
>>>>> That's because some client is trying to connect to the server, but the server is looking (not following or leading) and the result of the nc run confirms it. I think you still have a connectivity problem. There is something preventing the servers from connecting to each other and it seems to be something related to your environment.
>>>>>
>>>>> -Flavio
>>>>>
>>>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>> wrote:
>>>>>>
>>>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble. Now getting different messages in my out logs:P
>>>>>>
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>>>>
>>>>>> $ ./zkServer.sh status
>>>>>> JMX enabled by default
>>>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>>>>> Error contacting service. It is probably not running.
>>>>>>
>>>>>>
>>>>>> $ echo mntr | nc localhost 2181
>>>>>> This ZooKeeper instance is not currently serving requests
>>>>>>
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>>
>>>>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>>>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>>>>> To: Barnett, Barry F
>>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>>
>>>>>> Barry,
>>>>>> looks like you want to go to the machine where Zk is running and check
>>>>>> with "netstat -an" or some other tool on which port Zk is really
>>>>>> running... unless this is a NAT issue ;-)
>>>>>>
>>>>>> Cheers,
>>>>>> --Jürgen
>>>>>>
>>>>>> On 04.02.2015 16:51, penguin wrote:
>>>>>>> $ nc -v remotehostname 2181
>>>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection refused
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Barry
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ________________________________
>>>>>> If you reply to this email, your message will be added to the discussion below:
>>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>> ________________________________
>>>>> If you reply to this email, your message will be added to the discussion below:
>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here< href="" target="_top" rel="nofollow" link="external">
>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml <http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>>>>
>>>>
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion below:
>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580909.html
> To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580912.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.
RE: New Zookeeper Installation - connection refused
Posted by penguin <ba...@wellsfargo.com>.
Ok, the client port change worked for my connection refused issue with instances on the same server. I now have 2 followers and 1 leader. So I’ve determined that zk can run 3 instances on the same physical machine with no connection issues. I was getting the same connection refused error when I had 3 separate machines running 1 instance of zk on each, trying to instantiate an ensemble. Should I still have different client ports in that scenario? I had the same client ports and it was failing miserably. I am not familiar with zkconf, but I will look at it.
Regards,
Barry
From: Michi Mutsuzaki [via zookeeper-user] [mailto:ml-node+s578899n7580909h27@n2.nabble.com]
Sent: Thursday, March 12, 2015 1:26 AM
To: Barnett, Barry F
Subject: Re: New Zookeeper Installation - connection refused
Yes that's even better :)
On Wed, Mar 11, 2015 at 10:22 PM, Patrick Hunt <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=0>> wrote:
> Consider zkconf? https://github.com/phunt/zkconf
>
> Patrick
>
> On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=1>> wrote:
>> The clientPort parameter needs to be different for each instance if
>> you are running them on a same box. Something like this should work:
>>
>> https://paste.apache.org/n59c?action=download
>>
>> On Wed, Mar 11, 2015 at 8:02 AM, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=2>> wrote:
>>> Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK instances setup on the same physical machine, so no need to worry about whether multiple physical machines can communicate from one to the other, right?
>>> My setup on the Linux sever is as follows:
>>>
>>> Directory for instance 1: /home/zk1
>>> Directory for instance 2: /home/zk2
>>> Directory for instance 3: /home/zk3
>>>
>>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2, zk3 for the other instances):
>>> # The number of milliseconds of each tick
>>> tickTime=2000
>>> # The number of ticks that the initial
>>> # synchronization phase can take
>>> initLimit=10
>>> # The number of ticks that can pass between
>>> # sending a request and getting an acknowledgement
>>> syncLimit=5
>>> # the directory where the snapshot is stored.
>>> # do not use /tmp for storage, /tmp here is just
>>> # example sakes.
>>> dataDir=/home/zk1/zookeeper-3.4.6/data
>>> dataLogDir=/home/zk1/zookeeper-3.4.6/data
>>> # the port at which the clients will connect
>>> clientPort=2181
>>> # the maximum number of client connections.
>>> maxClientCnxns=60
>>> #
>>> # Be sure to read the maintenance section of the
>>> # administrator guide before turning on autopurge.
>>> #
>>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>> #
>>> # The number of snapshots to retain in dataDir
>>> #autopurge.snapRetainCount=3
>>> # Purge task interval in hours
>>> # Set to "0" to disable auto purge feature
>>> #autopurge.purgeInterval=1
>>> server.6=localhost:2888:3888
>>> server.7=localhost:2889:3889
>>> server.8=localhost:2890:3890
>>>
>>> the myid files are 6, 7 and 8 respectively for each instance. Is there something I'm missing because I'm still getting the connection refused warning, and the status of the zk servers doesn't show anything but an error.
>>>
>>> Zookeeper.out shows:
>>> 2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=6]:QuorumCnxManager@38
>>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>>> :339)
>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>>> pl.java:200)
>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>>> 82)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:579)
>>>
>>> zkServer.sh status shows:
>>>
>>> $ ./zkServer.sh status
>>> JMX enabled by default
>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>> Error contacting service. It is probably not running.
>>> $ ./zkServer.sh start
>>> JMX enabled by default
>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>> Starting zookeeper ... already running as process 27102.
>>>
>>> Regards,
>>>
>>> Barry
>>>
>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580909&i=3>]
>>> Sent: Tuesday, February 24, 2015 6:53 PM
>>> To: Barnett, Barry F
>>> Subject: Re: New Zookeeper Installation - connection refused
>>>
>>> It's hard to say what the problem is without more information. For leader election, the servers will try to create a TCP connection with each other, so if you have tested separately and your sample code worked for the same ports, the only thing I can think of is that there is a problem with the server name resolution. Is it possible that the ZK servers aren't resolving the names of your servers appropriately?
>>>
>>> Perhaps you could also use tcpdump or some similar tool to inspect the network traffic. In fact, I think someone here has written a tool to inspect the leader election traffic and display it. Was it Raul?
>>>
>>> -Flavio
>>>
>>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>> wrote:
>>>>
>>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither work, both receiving the 'connection refused' error.
>>>>
>>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>>>
>>>> Myid file is set to 1,2 and 3 respectively on the machines.
>>>> Zoo.cfg has the following, and uses 2181 as its client port:
>>>>
>>>> server.1=servername:2888:3888
>>>> server.2=servername:2888:3888
>>>> server.3=servername:2888:3888
>>>>
>>>> Interestingly enough, when I start up the server1 instance, the zookeeper.out file receives ongoing information on the connection refused to each server instance. When I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information. Is that normal?
>>>>
>>>> In any case, I've checked the ports by using nc -v hostname port, and connections are successful. So this is not a firewall issue or port block issue. I've had my OSE's check the servers as well to ensure no communications issue.
>>>>
>>>> *Note, port 2181 is an eForward port. Is this supposed to be used by Zookeeper?
>>>>
>>>> Same errors over and over:
>>>> 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING
>>>> 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815] - New e
>>>> lection. My id = 1, proposed zxid=0x0
>>>> 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification:
>>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
>>>> (n.peerEpoch) LOOKING (my state)
>>>> 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channe
>>>> l to 2 at election address servername:3888
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1> <mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2>>]
>>>> Sent: Wednesday, February 04, 2015 4:17 PM
>>>> To: Barnett, Barry F
>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>
>>>> That's because some client is trying to connect to the server, but the server is looking (not following or leading) and the result of the nc run confirms it. I think you still have a connectivity problem. There is something preventing the servers from connecting to each other and it seems to be something related to your environment.
>>>>
>>>> -Flavio
>>>>
>>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>> wrote:
>>>>>
>>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble. Now getting different messages in my out logs:P
>>>>>
>>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>>>
>>>>> $ ./zkServer.sh status
>>>>> JMX enabled by default
>>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>>>> Error contacting service. It is probably not running.
>>>>>
>>>>>
>>>>> $ echo mntr | nc localhost 2181
>>>>> This ZooKeeper instance is not currently serving requests
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Barry
>>>>>
>>>>>
>>>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>>>> To: Barnett, Barry F
>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>
>>>>> Barry,
>>>>> looks like you want to go to the machine where Zk is running and check
>>>>> with "netstat -an" or some other tool on which port Zk is really
>>>>> running... unless this is a NAT issue ;-)
>>>>>
>>>>> Cheers,
>>>>> --Jürgen
>>>>>
>>>>> On 04.02.2015 16:51, penguin wrote:
>>>>>> $ nc -v remotehostname 2181
>>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection refused
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> If you reply to this email, your message will be added to the discussion below:
>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion below:
>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here< href="" target="_top" rel="nofollow" link="external">
>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml <http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>>>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>>>
>>>
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
________________________________
If you reply to this email, your message will be added to the discussion below:
http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580909.html
To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580912.html
Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: New Zookeeper Installation - connection refused
Posted by Michi Mutsuzaki <mi...@cs.stanford.edu>.
Yes that's even better :)
On Wed, Mar 11, 2015 at 10:22 PM, Patrick Hunt <ph...@apache.org> wrote:
> Consider zkconf? https://github.com/phunt/zkconf
>
> Patrick
>
> On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki <mi...@cs.stanford.edu> wrote:
>> The clientPort parameter needs to be different for each instance if
>> you are running them on a same box. Something like this should work:
>>
>> https://paste.apache.org/n59c?action=download
>>
>> On Wed, Mar 11, 2015 at 8:02 AM, penguin <ba...@wellsfargo.com> wrote:
>>> Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK instances setup on the same physical machine, so no need to worry about whether multiple physical machines can communicate from one to the other, right?
>>> My setup on the Linux sever is as follows:
>>>
>>> Directory for instance 1: /home/zk1
>>> Directory for instance 2: /home/zk2
>>> Directory for instance 3: /home/zk3
>>>
>>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2, zk3 for the other instances):
>>> # The number of milliseconds of each tick
>>> tickTime=2000
>>> # The number of ticks that the initial
>>> # synchronization phase can take
>>> initLimit=10
>>> # The number of ticks that can pass between
>>> # sending a request and getting an acknowledgement
>>> syncLimit=5
>>> # the directory where the snapshot is stored.
>>> # do not use /tmp for storage, /tmp here is just
>>> # example sakes.
>>> dataDir=/home/zk1/zookeeper-3.4.6/data
>>> dataLogDir=/home/zk1/zookeeper-3.4.6/data
>>> # the port at which the clients will connect
>>> clientPort=2181
>>> # the maximum number of client connections.
>>> maxClientCnxns=60
>>> #
>>> # Be sure to read the maintenance section of the
>>> # administrator guide before turning on autopurge.
>>> #
>>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>> #
>>> # The number of snapshots to retain in dataDir
>>> #autopurge.snapRetainCount=3
>>> # Purge task interval in hours
>>> # Set to "0" to disable auto purge feature
>>> #autopurge.purgeInterval=1
>>> server.6=localhost:2888:3888
>>> server.7=localhost:2889:3889
>>> server.8=localhost:2890:3890
>>>
>>> the myid files are 6, 7 and 8 respectively for each instance. Is there something I'm missing because I'm still getting the connection refused warning, and the status of the zk servers doesn't show anything but an error.
>>>
>>> Zookeeper.out shows:
>>> 2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=6]:QuorumCnxManager@38
>>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>>> :339)
>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>>> pl.java:200)
>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>>> 82)
>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>>> at java.net.Socket.connect(Socket.java:579)
>>>
>>> zkServer.sh status shows:
>>>
>>> $ ./zkServer.sh status
>>> JMX enabled by default
>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>> Error contacting service. It is probably not running.
>>> $ ./zkServer.sh start
>>> JMX enabled by default
>>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>>> Starting zookeeper ... already running as process 27102.
>>>
>>> Regards,
>>>
>>> Barry
>>>
>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:ml-node+s578899n7580878h66@n2.nabble.com]
>>> Sent: Tuesday, February 24, 2015 6:53 PM
>>> To: Barnett, Barry F
>>> Subject: Re: New Zookeeper Installation - connection refused
>>>
>>> It's hard to say what the problem is without more information. For leader election, the servers will try to create a TCP connection with each other, so if you have tested separately and your sample code worked for the same ports, the only thing I can think of is that there is a problem with the server name resolution. Is it possible that the ZK servers aren't resolving the names of your servers appropriately?
>>>
>>> Perhaps you could also use tcpdump or some similar tool to inspect the network traffic. In fact, I think someone here has written a tool to inspect the leader election traffic and display it. Was it Raul?
>>>
>>> -Flavio
>>>
>>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>> wrote:
>>>>
>>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither work, both receiving the 'connection refused' error.
>>>>
>>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>>>
>>>> Myid file is set to 1,2 and 3 respectively on the machines.
>>>> Zoo.cfg has the following, and uses 2181 as its client port:
>>>>
>>>> server.1=servername:2888:3888
>>>> server.2=servername:2888:3888
>>>> server.3=servername:2888:3888
>>>>
>>>> Interestingly enough, when I start up the server1 instance, the zookeeper.out file receives ongoing information on the connection refused to each server instance. When I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information. Is that normal?
>>>>
>>>> In any case, I've checked the ports by using nc -v hostname port, and connections are successful. So this is not a firewall issue or port block issue. I've had my OSE's check the servers as well to ensure no communications issue.
>>>>
>>>> *Note, port 2181 is an eForward port. Is this supposed to be used by Zookeeper?
>>>>
>>>> Same errors over and over:
>>>> 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING
>>>> 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815] - New e
>>>> lection. My id = 1, proposed zxid=0x0
>>>> 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification:
>>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
>>>> (n.peerEpoch) LOOKING (my state)
>>>> 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channe
>>>> l to 2 at election address servername:3888
>>>> java.net.ConnectException: Connection refused
>>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1> <mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2>>]
>>>> Sent: Wednesday, February 04, 2015 4:17 PM
>>>> To: Barnett, Barry F
>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>
>>>> That's because some client is trying to connect to the server, but the server is looking (not following or leading) and the result of the nc run confirms it. I think you still have a connectivity problem. There is something preventing the servers from connecting to each other and it seems to be something related to your environment.
>>>>
>>>> -Flavio
>>>>
>>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>> wrote:
>>>>>
>>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble. Now getting different messages in my out logs:P
>>>>>
>>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>>>
>>>>> $ ./zkServer.sh status
>>>>> JMX enabled by default
>>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>>>> Error contacting service. It is probably not running.
>>>>>
>>>>>
>>>>> $ echo mntr | nc localhost 2181
>>>>> This ZooKeeper instance is not currently serving requests
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Barry
>>>>>
>>>>>
>>>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>>>> To: Barnett, Barry F
>>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>>
>>>>> Barry,
>>>>> looks like you want to go to the machine where Zk is running and check
>>>>> with "netstat -an" or some other tool on which port Zk is really
>>>>> running... unless this is a NAT issue ;-)
>>>>>
>>>>> Cheers,
>>>>> --Jürgen
>>>>>
>>>>> On 04.02.2015 16:51, penguin wrote:
>>>>>> $ nc -v remotehostname 2181
>>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection refused
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Barry
>>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>> If you reply to this email, your message will be added to the discussion below:
>>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion below:
>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here< href="" target="_top" rel="nofollow" link="external">
>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml <http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>>>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>>>
>>>
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: New Zookeeper Installation - connection refused
Posted by Patrick Hunt <ph...@apache.org>.
Consider zkconf? https://github.com/phunt/zkconf
Patrick
On Wed, Mar 11, 2015 at 10:21 PM, Michi Mutsuzaki <mi...@cs.stanford.edu> wrote:
> The clientPort parameter needs to be different for each instance if
> you are running them on a same box. Something like this should work:
>
> https://paste.apache.org/n59c?action=download
>
> On Wed, Mar 11, 2015 at 8:02 AM, penguin <ba...@wellsfargo.com> wrote:
>> Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK instances setup on the same physical machine, so no need to worry about whether multiple physical machines can communicate from one to the other, right?
>> My setup on the Linux sever is as follows:
>>
>> Directory for instance 1: /home/zk1
>> Directory for instance 2: /home/zk2
>> Directory for instance 3: /home/zk3
>>
>> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2, zk3 for the other instances):
>> # The number of milliseconds of each tick
>> tickTime=2000
>> # The number of ticks that the initial
>> # synchronization phase can take
>> initLimit=10
>> # The number of ticks that can pass between
>> # sending a request and getting an acknowledgement
>> syncLimit=5
>> # the directory where the snapshot is stored.
>> # do not use /tmp for storage, /tmp here is just
>> # example sakes.
>> dataDir=/home/zk1/zookeeper-3.4.6/data
>> dataLogDir=/home/zk1/zookeeper-3.4.6/data
>> # the port at which the clients will connect
>> clientPort=2181
>> # the maximum number of client connections.
>> maxClientCnxns=60
>> #
>> # Be sure to read the maintenance section of the
>> # administrator guide before turning on autopurge.
>> #
>> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>> #
>> # The number of snapshots to retain in dataDir
>> #autopurge.snapRetainCount=3
>> # Purge task interval in hours
>> # Set to "0" to disable auto purge feature
>> #autopurge.purgeInterval=1
>> server.6=localhost:2888:3888
>> server.7=localhost:2889:3889
>> server.8=localhost:2890:3890
>>
>> the myid files are 6, 7 and 8 respectively for each instance. Is there something I'm missing because I'm still getting the connection refused warning, and the status of the zk servers doesn't show anything but an error.
>>
>> Zookeeper.out shows:
>> 2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=6]:QuorumCnxManager@38
>> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
>> java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
>> :339)
>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
>> pl.java:200)
>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
>> 82)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>> at java.net.Socket.connect(Socket.java:579)
>>
>> zkServer.sh status shows:
>>
>> $ ./zkServer.sh status
>> JMX enabled by default
>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>> Error contacting service. It is probably not running.
>> $ ./zkServer.sh start
>> JMX enabled by default
>> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
>> Starting zookeeper ... already running as process 27102.
>>
>> Regards,
>>
>> Barry
>>
>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:ml-node+s578899n7580878h66@n2.nabble.com]
>> Sent: Tuesday, February 24, 2015 6:53 PM
>> To: Barnett, Barry F
>> Subject: Re: New Zookeeper Installation - connection refused
>>
>> It's hard to say what the problem is without more information. For leader election, the servers will try to create a TCP connection with each other, so if you have tested separately and your sample code worked for the same ports, the only thing I can think of is that there is a problem with the server name resolution. Is it possible that the ZK servers aren't resolving the names of your servers appropriately?
>>
>> Perhaps you could also use tcpdump or some similar tool to inspect the network traffic. In fact, I think someone here has written a tool to inspect the leader election traffic and display it. Was it Raul?
>>
>> -Flavio
>>
>>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>> wrote:
>>>
>>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither work, both receiving the 'connection refused' error.
>>>
>>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>>
>>> Myid file is set to 1,2 and 3 respectively on the machines.
>>> Zoo.cfg has the following, and uses 2181 as its client port:
>>>
>>> server.1=servername:2888:3888
>>> server.2=servername:2888:3888
>>> server.3=servername:2888:3888
>>>
>>> Interestingly enough, when I start up the server1 instance, the zookeeper.out file receives ongoing information on the connection refused to each server instance. When I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information. Is that normal?
>>>
>>> In any case, I've checked the ports by using nc -v hostname port, and connections are successful. So this is not a firewall issue or port block issue. I've had my OSE's check the servers as well to ensure no communications issue.
>>>
>>> *Note, port 2181 is an eForward port. Is this supposed to be used by Zookeeper?
>>>
>>> Same errors over and over:
>>> 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING
>>> 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815] - New e
>>> lection. My id = 1, proposed zxid=0x0
>>> 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification:
>>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
>>> (n.peerEpoch) LOOKING (my state)
>>> 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channe
>>> l to 2 at election address servername:3888
>>> java.net.ConnectException: Connection refused
>>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>>
>>> Regards,
>>>
>>> Barry
>>>
>>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1> <mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2>>]
>>> Sent: Wednesday, February 04, 2015 4:17 PM
>>> To: Barnett, Barry F
>>> Subject: Re: New Zookeeper Installation - connection refused
>>>
>>> That's because some client is trying to connect to the server, but the server is looking (not following or leading) and the result of the nc run confirms it. I think you still have a connectivity problem. There is something preventing the servers from connecting to each other and it seems to be something related to your environment.
>>>
>>> -Flavio
>>>
>>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>> wrote:
>>>>
>>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble. Now getting different messages in my out logs:P
>>>>
>>>> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>>> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>>
>>>> $ ./zkServer.sh status
>>>> JMX enabled by default
>>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>>> Error contacting service. It is probably not running.
>>>>
>>>>
>>>> $ echo mntr | nc localhost 2181
>>>> This ZooKeeper instance is not currently serving requests
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>>
>>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>>> To: Barnett, Barry F
>>>> Subject: Re: New Zookeeper Installation - connection refused
>>>>
>>>> Barry,
>>>> looks like you want to go to the machine where Zk is running and check
>>>> with "netstat -an" or some other tool on which port Zk is really
>>>> running... unless this is a NAT issue ;-)
>>>>
>>>> Cheers,
>>>> --Jürgen
>>>>
>>>> On 04.02.2015 16:51, penguin wrote:
>>>>> $ nc -v remotehostname 2181
>>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection refused
>>>>>
>>>>> Regards,
>>>>>
>>>>> Barry
>>>>>
>>>>
>>>>
>>>> ________________________________
>>>> If you reply to this email, your message will be added to the discussion below:
>>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>>
>>>
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>>> To unsubscribe from New Zookeeper Installation - connection refused, click here< href="" target="_top" rel="nofollow" link="external">
>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml <http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
>> To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>>
>>
>> --
>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
>> Sent from the zookeeper-user mailing list archive at Nabble.com.
Re: New Zookeeper Installation - connection refused
Posted by Michi Mutsuzaki <mi...@cs.stanford.edu>.
The clientPort parameter needs to be different for each instance if
you are running them on a same box. Something like this should work:
https://paste.apache.org/n59c?action=download
On Wed, Mar 11, 2015 at 8:02 AM, penguin <ba...@wellsfargo.com> wrote:
> Ok, I have setup my ZooKeeper ensemble on the same server. I have 3 ZK instances setup on the same physical machine, so no need to worry about whether multiple physical machines can communicate from one to the other, right?
> My setup on the Linux sever is as follows:
>
> Directory for instance 1: /home/zk1
> Directory for instance 2: /home/zk2
> Directory for instance 3: /home/zk3
>
> In my conf directory, I have my zoo.cfg file for each (replace zk1 with zk2, zk3 for the other instances):
> # The number of milliseconds of each tick
> tickTime=2000
> # The number of ticks that the initial
> # synchronization phase can take
> initLimit=10
> # The number of ticks that can pass between
> # sending a request and getting an acknowledgement
> syncLimit=5
> # the directory where the snapshot is stored.
> # do not use /tmp for storage, /tmp here is just
> # example sakes.
> dataDir=/home/zk1/zookeeper-3.4.6/data
> dataLogDir=/home/zk1/zookeeper-3.4.6/data
> # the port at which the clients will connect
> clientPort=2181
> # the maximum number of client connections.
> maxClientCnxns=60
> #
> # Be sure to read the maintenance section of the
> # administrator guide before turning on autopurge.
> #
> # http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
> #
> # The number of snapshots to retain in dataDir
> #autopurge.snapRetainCount=3
> # Purge task interval in hours
> # Set to "0" to disable auto purge feature
> #autopurge.purgeInterval=1
> server.6=localhost:2888:3888
> server.7=localhost:2889:3889
> server.8=localhost:2890:3890
>
> the myid files are 6, 7 and 8 respectively for each instance. Is there something I'm missing because I'm still getting the connection refused warning, and the status of the zk servers doesn't show anything but an error.
>
> Zookeeper.out shows:
> 2015-03-11 08:59:22,238 [myid:6] - WARN [WorkerSender[myid=6]:QuorumCnxManager@38
> 2] - Cannot open channel to 8 at election address localhost/127.0.0.1:3890
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java
> :339)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketIm
> pl.java:200)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:1
> 82)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
>
> zkServer.sh status shows:
>
> $ ./zkServer.sh status
> JMX enabled by default
> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
> Error contacting service. It is probably not running.
> $ ./zkServer.sh start
> JMX enabled by default
> Using config: /opt/apps/zk1/zookeeper-3.4.6/bin/../conf/zoo.cfg
> Starting zookeeper ... already running as process 27102.
>
> Regards,
>
> Barry
>
> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:ml-node+s578899n7580878h66@n2.nabble.com]
> Sent: Tuesday, February 24, 2015 6:53 PM
> To: Barnett, Barry F
> Subject: Re: New Zookeeper Installation - connection refused
>
> It's hard to say what the problem is without more information. For leader election, the servers will try to create a TCP connection with each other, so if you have tested separately and your sample code worked for the same ports, the only thing I can think of is that there is a problem with the server name resolution. Is it possible that the ZK servers aren't resolving the names of your servers appropriately?
>
> Perhaps you could also use tcpdump or some similar tool to inspect the network traffic. In fact, I think someone here has written a tool to inspect the leader election traffic and display it. Was it Raul?
>
> -Flavio
>
>> On 24 Feb 2015, at 16:31, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=0>> wrote:
>>
>> Ok, I have tried various setups (5 and 3 ZK Server instances), and neither work, both receiving the 'connection refused' error.
>>
>> I'm now at the 3 ZK ensemble, 1 ZK server instance per physical machine.
>>
>> Myid file is set to 1,2 and 3 respectively on the machines.
>> Zoo.cfg has the following, and uses 2181 as its client port:
>>
>> server.1=servername:2888:3888
>> server.2=servername:2888:3888
>> server.3=servername:2888:3888
>>
>> Interestingly enough, when I start up the server1 instance, the zookeeper.out file receives ongoing information on the connection refused to each server instance. When I start up the servers on machines 2 and 3, those zookeeper.out files do not receive any information. Is that normal?
>>
>> In any case, I've checked the ports by using nc -v hostname port, and connections are successful. So this is not a firewall issue or port block issue. I've had my OSE's check the servers as well to ensure no communications issue.
>>
>> *Note, port 2181 is an eForward port. Is this supposed to be used by Zookeeper?
>>
>> Same errors over and over:
>> 2015-02-24 09:38:00,601 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:QuorumPeer@714] - LOOKING
>> 2015-02-24 09:38:00,603 [myid:1] - INFO [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@815] - New e
>> lection. My id = 1, proposed zxid=0x0
>> 2015-02-24 09:38:00,605 [myid:1] - INFO [WorkerReceiver[myid=1]:FastLeaderElection@597] - Notification:
>> 1 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0
>> (n.peerEpoch) LOOKING (my state)
>> 2015-02-24 09:38:00,608 [myid:1] - WARN [WorkerSender[myid=1]:QuorumCnxManager@382] - Cannot open channe
>> l to 2 at election address servername:3888
>> java.net.ConnectException: Connection refused
>> at java.net.PlainSocketImpl.socketConnect(Native Method)
>> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>>
>> Regards,
>>
>> Barry
>>
>> From: Flavio Junqueira-2 [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=1> <mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580878&i=2>>]
>> Sent: Wednesday, February 04, 2015 4:17 PM
>> To: Barnett, Barry F
>> Subject: Re: New Zookeeper Installation - connection refused
>>
>> That's because some client is trying to connect to the server, but the server is looking (not following or leading) and the result of the nc run confirms it. I think you still have a connectivity problem. There is something preventing the servers from connecting to each other and it seems to be something related to your environment.
>>
>> -Flavio
>>
>>> On 04 Feb 2015, at 17:51, penguin <[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=0>> wrote:
>>>
>>> I now made this a 3 server ensemble as opposed to a 5 server ensemble. Now getting different messages in my out logs:P
>>>
>>> 2015-02-04 11:48:29,851 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@362] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
>>> 2015-02-04 11:48:29,851 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /xxxxxxxx:64089 (no session established for client)
>>>
>>> $ ./zkServer.sh status
>>> JMX enabled by default
>>> Using config: /opt/apps/current/bin/../conf/zoo.cfg
>>> Error contacting service. It is probably not running.
>>>
>>>
>>> $ echo mntr | nc localhost 2181
>>> This ZooKeeper instance is not currently serving requests
>>>
>>>
>>>
>>> Regards,
>>>
>>> Barry
>>>
>>>
>>> From: "Jürgen Wagner (DVT)" [via zookeeper-user] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=7580844&i=1>]
>>> Sent: Wednesday, February 04, 2015 10:56 AM
>>> To: Barnett, Barry F
>>> Subject: Re: New Zookeeper Installation - connection refused
>>>
>>> Barry,
>>> looks like you want to go to the machine where Zk is running and check
>>> with "netstat -an" or some other tool on which port Zk is really
>>> running... unless this is a NAT issue ;-)
>>>
>>> Cheers,
>>> --Jürgen
>>>
>>> On 04.02.2015 16:51, penguin wrote:
>>>> $ nc -v remotehostname 2181
>>>> nc: connect to remotehostname port 2181 (tcp) failed: Connection refused
>>>>
>>>> Regards,
>>>>
>>>> Barry
>>>>
>>>
>>>
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion below:
>>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580838.html>
>>> To unsubscribe from New Zookeeper Installation - connection refused, click here<
>>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580840.html
>>> Sent from the zookeeper-user mailing list archive at Nabble.com.
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion below:
>> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580844.html>
>> To unsubscribe from New Zookeeper Installation - connection refused, click here< href="" target="_top" rel="nofollow" link="external">
>> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml <http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>>
>>
>>
>>
>>
>> --
>> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html <http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580875.html>
>> Sent from the zookeeper-user mailing list archive at Nabble.com <http://nabble.com/>.
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580878.html
> To unsubscribe from New Zookeeper Installation - connection refused, click here<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=7580825&code=YmFycnkuYmFybmV0dEB3ZWxsc2ZhcmdvLmNvbXw3NTgwODI1fC0xNDYzMjU3NTMx>.
> NAML<http://zookeeper-user.578899.n2.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
>
>
> --
> View this message in context: http://zookeeper-user.578899.n2.nabble.com/New-Zookeeper-Installation-connection-refused-tp7580825p7580904.html
> Sent from the zookeeper-user mailing list archive at Nabble.com.