You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by "Mahadev konar (JIRA)" <ji...@apache.org> on 2008/07/23 20:29:33 UTC
[jira] Created: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Follower does not shut itself down if its too far behind the leader.
--------------------------------------------------------------------
Key: ZOOKEEPER-87
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
Project: Zookeeper
Issue Type: Bug
Reporter: Mahadev konar
Assignee: Mahadev konar
Priority: Critical
Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-87:
-----------------------------------
Fix Version/s: 3.0.0
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.0.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Vishal K (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921355#action_12921355 ]
Vishal K commented on ZOOKEEPER-87:
-----------------------------------
Hi Mahadev, Flavio,
I am trying to understand how shutting down the follower will help in this case.
Mahadev can you please explain what you man by : "The follower should shut itself down if it is not able to keep up to the leader within some limit so that guarantee of updates can be made to the clients connected to different servers."
Under what conditions do you expect a shutdown of follower handler to eliminate the problem of follower lagging behind. Are you referring to clients connected to this server that will fail-over their session to another server?
Thanks.
-Vishal
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Components: quorum
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.4.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Flavio Paiva Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616556#action_12616556 ]
Flavio Paiva Junqueira commented on ZOOKEEPER-87:
-------------------------------------------------
If you throttle at the follower, then the queue on the leader side will build up, and you won't have the problem you mention. The approach I describe seems to be safer as you don't have to wait for the next Leader.PING period to make a decision. It is true that the leader can send a ping at any time it wants, but if the leader is clueless as you say, then it will wait for the next ping period and it won't send any exceptional pings.
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.0.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616566#action_12616566 ]
Mahadev konar commented on ZOOKEEPER-87:
----------------------------------------
throttling at the follower is a little tricky. do you mean the follower will stop reading requests ?
since you do not know what is on the tcp stack ---
when you do a send it might be buffered on either of the tcp stacks on the leader and the follower. so you do not know how many txn's are buffered up the tcp stack. So throttling is a tricky thing with the follower and leader. also the rsponse to leader.PING are critical with session management.
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.0.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-87:
----------------------------------
Fix Version/s: (was: 3.0.0)
3.1.0
Assignee: (was: Mahadev konar)
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Mahadev konar
> Priority: Critical
> Fix For: 3.1.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616550#action_12616550 ]
Mahadev konar commented on ZOOKEEPER-87:
----------------------------------------
i was thinking abt the same lines but not on maximum capacity on follower handlers --
ben and I had a discussion regarding this and we decided on doing htis ---
1) we send a Leader.PING from the leader to the follower --- we will include the ticktime at the leader and send it to the follower
2) the follower just returns back the tick time with the ping response.
3) we update the ticktime that we saw last from the follower and then kill the follower handler if its lagging behind.
-- this approach is less error prone since we are getting response from the follower and know whats the last ticktime it processed. The problem with your suggestion is that you might be sending it out fast to the follower but the follower might be buffering all of it and not doing anything with it...
though the above suggestion has a problem right now since the Leader.PING does not go through the pipeline at the follower. so a response to the ping does not necessarily mean that the follower is in line with processing the requests. We have to get an end to end response from the follower to see how much it is behind..
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.0.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated ZOOKEEPER-87:
-----------------------------------
Fix Version/s: (was: 3.1.0)
3.2.0
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-87:
----------------------------------
Component/s: quorum
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Components: quorum
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.2.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925165#action_12925165 ]
Mahadev konar commented on ZOOKEEPER-87:
----------------------------------------
exactly vishal. The problem is to avoid clients being connected to a server which can lag behind the leader.
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Components: quorum
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.4.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Flavio Paiva Junqueira (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616435#action_12616435 ]
Flavio Paiva Junqueira commented on ZOOKEEPER-87:
-------------------------------------------------
Mahadev, I was thinking that we we could set a maximum capacity to queuedPackets on FollowerHandler, and check the remaining capacity on FollowerHandler.queuePacket() before accepting a new packet. We could then invoke FollowerHandler.shutdown() in the case we reach the capacity. What do you think? If you think this is a good idea, I can create a patch.
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.0.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar reassigned ZOOKEEPER-87:
--------------------------------------
Assignee: Mahadev konar
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.1.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-87) Follower does not shut itself down
if its too far behind the leader.
Posted by "Patrick Hunt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-87?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated ZOOKEEPER-87:
----------------------------------
Fix Version/s: (was: 3.2.0)
3.3.0
Release Note: not a blocker for 3.2, moving to 3.3
> Follower does not shut itself down if its too far behind the leader.
> --------------------------------------------------------------------
>
> Key: ZOOKEEPER-87
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-87
> Project: Zookeeper
> Issue Type: Bug
> Components: quorum
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Critical
> Fix For: 3.3.0
>
>
> Currently, the follower if lagging behind keeps sending pings to the leader it will stay alive and will keep getting further and further behind the leader. The follower should shut itself down if it is not able to keep up to the leader within some limit so that gurantee of updates can be made to the clients connected to different servers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.