You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Vishal K (JIRA)" <ji...@apache.org> on 2010/08/16 16:13:19 UTC

[jira] Created: (ZOOKEEPER-851) ZK lets any node to become an observer

ZK lets any node to become an observer
--------------------------------------

                 Key: ZOOKEEPER-851
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-851
             Project: Zookeeper
          Issue Type: Bug
            Reporter: Vishal K
             Fix For: 3.3.1


I had a 3 node cluster running. The zoo.cfg on each contained 3 entries as show below:

tickTime=2000
dataDir=/var/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.0=10.150.27.61:2888:3888
server.1=10.150.27.62:2888:3888
server.2=10.150.27.63:2888:3888

I wanted to add another node to the cluster. In fourth node's zoo.cfg, I created another entry for that node and started zk server. The zoo.cfg on the first 3 nodes was left unchanged. The fourth node was able to join the cluster even though the 3 nodes had no idea about the fourth node.

zoo.cfg on fourth node:
tickTime=2000
dataDir=/var/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.0=10.150.27.61:2888:3888
server.1=10.150.27.62:2888:3888
server.2=10.150.27.63:2888:3888
server.3=10.17.117.71:2888:3888

It looks like 10.17.117.71 is becoming an observer in this case. I was expecting that the leader will reject 10.17.117.71.

# telnet 10.17.117.71 2181
Trying 10.17.117.71...
Connected to 10.17.117.71.
Escape character is '^]'.
stat
Zookeeper version: 3.3.0--1, built on 04/02/2010 22:40 GMT
Clients:
 /10.17.117.71:37297[1](queued=0,recved=1,sent=0)

Latency min/avg/max: 0/0/0
Received: 3
Sent: 2
Outstanding: 0
Zxid: 0x200000065
Mode: follower
Node count: 288

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (ZOOKEEPER-851) ZK lets any node to become an observer

Posted by "Vishal K (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/ZOOKEEPER-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921340#action_12921340 ] 

Vishal K commented on ZOOKEEPER-851:
------------------------------------

Hi Henry ,

Any comments?

Thanks.
-Vishal

> ZK lets any node to become an observer
> --------------------------------------
>
>                 Key: ZOOKEEPER-851
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-851
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: quorum, server
>    Affects Versions: 3.3.1
>            Reporter: Vishal K
>            Priority: Critical
>             Fix For: 3.4.0
>
>
> I had a 3 node cluster running. The zoo.cfg on each contained 3 entries as show below:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> I wanted to add another node to the cluster. In fourth node's zoo.cfg, I created another entry for that node and started zk server. The zoo.cfg on the first 3 nodes was left unchanged. The fourth node was able to join the cluster even though the 3 nodes had no idea about the fourth node.
> zoo.cfg on fourth node:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> server.3=10.17.117.71:2888:3888
> It looks like 10.17.117.71 is becoming an observer in this case. I was expecting that the leader will reject 10.17.117.71.
> # telnet 10.17.117.71 2181
> Trying 10.17.117.71...
> Connected to 10.17.117.71.
> Escape character is '^]'.
> stat
> Zookeeper version: 3.3.0--1, built on 04/02/2010 22:40 GMT
> Clients:
>  /10.17.117.71:37297[1](queued=0,recved=1,sent=0)
> Latency min/avg/max: 0/0/0
> Received: 3
> Sent: 2
> Outstanding: 0
> Zxid: 0x200000065
> Mode: follower
> Node count: 288

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (ZOOKEEPER-851) ZK lets any node to become an observer

Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/ZOOKEEPER-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925927#action_12925927 ] 

Henry Robinson commented on ZOOKEEPER-851:
------------------------------------------

Hi Vishal - 

Sorry for the slow turnaround on this one. It doesn't surprise me that this is the behaviour, although it's slightly unexpected that the node becomes an observer, rather than a follower. What evidence do you have for that? (Given that Mode: follower - I haven't checked the code in a while, but I would have thought it would print Mode: Observer).

Henry

> ZK lets any node to become an observer
> --------------------------------------
>
>                 Key: ZOOKEEPER-851
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-851
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: quorum, server
>    Affects Versions: 3.3.1
>            Reporter: Vishal K
>            Priority: Critical
>             Fix For: 3.4.0
>
>
> I had a 3 node cluster running. The zoo.cfg on each contained 3 entries as show below:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> I wanted to add another node to the cluster. In fourth node's zoo.cfg, I created another entry for that node and started zk server. The zoo.cfg on the first 3 nodes was left unchanged. The fourth node was able to join the cluster even though the 3 nodes had no idea about the fourth node.
> zoo.cfg on fourth node:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> server.3=10.17.117.71:2888:3888
> It looks like 10.17.117.71 is becoming an observer in this case. I was expecting that the leader will reject 10.17.117.71.
> # telnet 10.17.117.71 2181
> Trying 10.17.117.71...
> Connected to 10.17.117.71.
> Escape character is '^]'.
> stat
> Zookeeper version: 3.3.0--1, built on 04/02/2010 22:40 GMT
> Clients:
>  /10.17.117.71:37297[1](queued=0,recved=1,sent=0)
> Latency min/avg/max: 0/0/0
> Received: 3
> Sent: 2
> Outstanding: 0
> Zxid: 0x200000065
> Mode: follower
> Node count: 288

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (ZOOKEEPER-851) ZK lets any node to become an observer

Posted by "Vishal K (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/ZOOKEEPER-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925997#action_12925997 ] 

Vishal K commented on ZOOKEEPER-851:
------------------------------------

I have not used observers yet. Looks like then I need to verify if this follower can become a leader.  I think I had tried this. If I remember, the 4th node did not become a leader. But this was a while back. I will try it again and update the jira.

> ZK lets any node to become an observer
> --------------------------------------
>
>                 Key: ZOOKEEPER-851
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-851
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: quorum, server
>    Affects Versions: 3.3.1
>            Reporter: Vishal K
>            Priority: Critical
>             Fix For: 3.4.0
>
>
> I had a 3 node cluster running. The zoo.cfg on each contained 3 entries as show below:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> I wanted to add another node to the cluster. In fourth node's zoo.cfg, I created another entry for that node and started zk server. The zoo.cfg on the first 3 nodes was left unchanged. The fourth node was able to join the cluster even though the 3 nodes had no idea about the fourth node.
> zoo.cfg on fourth node:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> server.3=10.17.117.71:2888:3888
> It looks like 10.17.117.71 is becoming an observer in this case. I was expecting that the leader will reject 10.17.117.71.
> # telnet 10.17.117.71 2181
> Trying 10.17.117.71...
> Connected to 10.17.117.71.
> Escape character is '^]'.
> stat
> Zookeeper version: 3.3.0--1, built on 04/02/2010 22:40 GMT
> Clients:
>  /10.17.117.71:37297[1](queued=0,recved=1,sent=0)
> Latency min/avg/max: 0/0/0
> Received: 3
> Sent: 2
> Outstanding: 0
> Zxid: 0x200000065
> Mode: follower
> Node count: 288

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (ZOOKEEPER-851) ZK lets any node to become an observer

Posted by "Henry Robinson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/ZOOKEEPER-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926016#action_12926016 ] 

Henry Robinson commented on ZOOKEEPER-851:
------------------------------------------

I think what happens is that the leader happily lets the new follower connect, but that it won't be part of any voting procedure. It shouldn't become leader because no other nodes know about it to  propose or support a vote for it. 

To add a new node, you'll need to incrementally restart every node in your cluster with the new config.

> ZK lets any node to become an observer
> --------------------------------------
>
>                 Key: ZOOKEEPER-851
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-851
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: quorum, server
>    Affects Versions: 3.3.1
>            Reporter: Vishal K
>            Priority: Critical
>             Fix For: 3.4.0
>
>
> I had a 3 node cluster running. The zoo.cfg on each contained 3 entries as show below:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> I wanted to add another node to the cluster. In fourth node's zoo.cfg, I created another entry for that node and started zk server. The zoo.cfg on the first 3 nodes was left unchanged. The fourth node was able to join the cluster even though the 3 nodes had no idea about the fourth node.
> zoo.cfg on fourth node:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> server.3=10.17.117.71:2888:3888
> It looks like 10.17.117.71 is becoming an observer in this case. I was expecting that the leader will reject 10.17.117.71.
> # telnet 10.17.117.71 2181
> Trying 10.17.117.71...
> Connected to 10.17.117.71.
> Escape character is '^]'.
> stat
> Zookeeper version: 3.3.0--1, built on 04/02/2010 22:40 GMT
> Clients:
>  /10.17.117.71:37297[1](queued=0,recved=1,sent=0)
> Latency min/avg/max: 0/0/0
> Received: 3
> Sent: 2
> Outstanding: 0
> Zxid: 0x200000065
> Mode: follower
> Node count: 288

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (ZOOKEEPER-851) ZK lets any node to become an observer

Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/ZOOKEEPER-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Patrick Hunt updated ZOOKEEPER-851:
-----------------------------------

        Fix Version/s: 3.4.0
                           (was: 3.3.1)
    Affects Version/s: 3.3.1
             Priority: Critical  (was: Major)
          Component/s: quorum
                       server

This does sound like incorrect behavior to me. Henry what do you think? slating for 3.4.0

> ZK lets any node to become an observer
> --------------------------------------
>
>                 Key: ZOOKEEPER-851
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-851
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: quorum, server
>    Affects Versions: 3.3.1
>            Reporter: Vishal K
>            Priority: Critical
>             Fix For: 3.4.0
>
>
> I had a 3 node cluster running. The zoo.cfg on each contained 3 entries as show below:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> I wanted to add another node to the cluster. In fourth node's zoo.cfg, I created another entry for that node and started zk server. The zoo.cfg on the first 3 nodes was left unchanged. The fourth node was able to join the cluster even though the 3 nodes had no idea about the fourth node.
> zoo.cfg on fourth node:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> server.3=10.17.117.71:2888:3888
> It looks like 10.17.117.71 is becoming an observer in this case. I was expecting that the leader will reject 10.17.117.71.
> # telnet 10.17.117.71 2181
> Trying 10.17.117.71...
> Connected to 10.17.117.71.
> Escape character is '^]'.
> stat
> Zookeeper version: 3.3.0--1, built on 04/02/2010 22:40 GMT
> Clients:
>  /10.17.117.71:37297[1](queued=0,recved=1,sent=0)
> Latency min/avg/max: 0/0/0
> Received: 3
> Sent: 2
> Outstanding: 0
> Zxid: 0x200000065
> Mode: follower
> Node count: 288

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (ZOOKEEPER-851) ZK lets any node to become an observer

Posted by "Vishal K (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/ZOOKEEPER-851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925170#action_12925170 ] 

Vishal K commented on ZOOKEEPER-851:
------------------------------------

going once, going twice ... :)

> ZK lets any node to become an observer
> --------------------------------------
>
>                 Key: ZOOKEEPER-851
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-851
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: quorum, server
>    Affects Versions: 3.3.1
>            Reporter: Vishal K
>            Priority: Critical
>             Fix For: 3.4.0
>
>
> I had a 3 node cluster running. The zoo.cfg on each contained 3 entries as show below:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> I wanted to add another node to the cluster. In fourth node's zoo.cfg, I created another entry for that node and started zk server. The zoo.cfg on the first 3 nodes was left unchanged. The fourth node was able to join the cluster even though the 3 nodes had no idea about the fourth node.
> zoo.cfg on fourth node:
> tickTime=2000
> dataDir=/var/zookeeper
> clientPort=2181
> initLimit=5
> syncLimit=2
> server.0=10.150.27.61:2888:3888
> server.1=10.150.27.62:2888:3888
> server.2=10.150.27.63:2888:3888
> server.3=10.17.117.71:2888:3888
> It looks like 10.17.117.71 is becoming an observer in this case. I was expecting that the leader will reject 10.17.117.71.
> # telnet 10.17.117.71 2181
> Trying 10.17.117.71...
> Connected to 10.17.117.71.
> Escape character is '^]'.
> stat
> Zookeeper version: 3.3.0--1, built on 04/02/2010 22:40 GMT
> Clients:
>  /10.17.117.71:37297[1](queued=0,recved=1,sent=0)
> Latency min/avg/max: 0/0/0
> Received: 3
> Sent: 2
> Outstanding: 0
> Zxid: 0x200000065
> Mode: follower
> Node count: 288

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.