[jira] [Created] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Created) (JIRA)" <ji...@apache.org>]

Add zk.updateServerList(newServerList) 
---------------------------------------

                 Key: ZOOKEEPER-1355
                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
             Project: ZooKeeper
          Issue Type: New Feature
          Components: java client
            Reporter: Alexander Shraer
            Assignee: Alexander Shraer
             Fix For: 3.5.0


When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.

It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.

We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491608#comment-13491608 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12552304/ZOOKEEPER-1355-6-Nov-ver2.patch
  against trunk revision 1404288.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1251//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1251//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1251//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Benjamin Reed (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184245#comment-13184245 ] 

Benjamin Reed commented on ZOOKEEPER-1355:
------------------------------------------

this looks really good! i have a couple of requests and something to think about:

1) you need to use spaces rather than tabs. you have a bunch of tabs in the patch. (there is a setting in eclipse to do tabs rather than spaces)
2) it would be nice in updateList to comment about the overall strategy for determining the return code
3) can you add a comment to "(currentIndex >= serverAddresses.size())" to say why >= is needed? i think i see it in the context of this patch, but later people might be tempted to change it back to ==

something to think about: on the updateServerList method, perhaps rather than making the second parameter a boolean we can make it the InetSocketAddress of the currently connected server. what do you think? i don't feel strongly about it.

                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496900#comment-13496900 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12553459/ZOOKEEPER-1355-13-Nov.patch
  against trunk revision 1404288.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1260//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1260//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1260//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Nov.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Flavio Junqueira (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497316#comment-13497316 ] 

Flavio Junqueira commented on ZOOKEEPER-1355:
---------------------------------------------

+1, thanks guys for the work. Is anyone else interested in reviewing this patch? If so, please comment asap. I'm fine with committing it myself before the end of the week or having someone else doing it, like Ben, since he pushed a lot for this patch.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Nov.patch, ZOOKEEPER-1355-13-Nov-ver2.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194308#comment-13194308 ] 

Patrick Hunt commented on ZOOKEEPER-1355:
-----------------------------------------

I think having that as a link would be useful. You could attach it on the CWIKI and put a link to that. Given it's a pdf it's going to be hard to make modifications going fwd though. Not sure if that can be helped but if it's an attachment on the wiki we could at least capture notes/changes there.

                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240054#comment-13240054 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12520208/ZOOKEEPER-1355-ver11.patch
  against trunk revision 1302736.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 34 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1016//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1016//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1016//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217027#comment-13217027 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12516134/ZOOKEEPER-1355-ver9.patch
  against trunk revision 1294000.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 8 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/962//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/962//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/962//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Benjamin Reed (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491673#comment-13491673 ] 

Benjamin Reed commented on ZOOKEEPER-1355:
------------------------------------------

michi and flavio are you guys good with this? please express concern or +1 :) if i don't hear anything by tomorrow, i'll assume the latter.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver10-3.patch

Disabling the forced disconnect on server reconfig as it's causing grief on first connect and causing other tests to fail.

Need to handle first connect more intelligently but don't have time to get to this today. 

Uploading a fixed version so unit tests will pass.

Will upload an improved version tomorrow that deals with first connection properly.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191826#comment-13191826 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12511612/ZOOKEEPER-1355-ver5.patch
  against trunk revision 1234974.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/915//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/915//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/915//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13243032#comment-13243032 ] 

Marshall McMullen commented on ZOOKEEPER-1355:
----------------------------------------------

I posted this latest patch on review board: https://reviews.apache.org/r/4596/
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver11.patch

This version fixes the core unit tests that were failing. 

All tests pass locally for me now.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Flavio Junqueira (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491926#comment-13491926 ] 

Flavio Junqueira commented on ZOOKEEPER-1355:
---------------------------------------------

Thanks Marshall and Alex for working on the patch. Is the latest patch on the review board? Until I check it on the review board it is -1 for me. ;-)
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-6-Nov.patch

Addresses all the concerns Michi had with the C client over on review board https://reviews.apache.org/r/3781/#review12460
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491569#comment-13491569 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12552299/ZOOKEEPER-1355-6-Nov.patch
  against trunk revision 1404288.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1250//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1250//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1250//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Benjamin Reed (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193999#comment-13193999 ] 

Benjamin Reed commented on ZOOKEEPER-1355:
------------------------------------------

i'll let alex provide a better answer, but just to test my understanding:

the scary math symbols boil down to a simple set of 4 rules for maintaining balanced load when the serverList changes. (the rules are at the end of the document) if you try to do something naive, like randomly disconnecting from the server you are connecting to, it will cause lots of reconnects and you still will end up with imbalanced load. it all boils down to four simple rules for determining when to disconnect from your current host. the math is just there so that it is clear that it wasn't pulled from thin air.

as far as the public api. we have to add the updateServerList since that is the whole point of the bug. i have no strong feelings about getCurrentHost() in terms of the name or making it public. it is only different from testableRemoteSocketAddress in the sense that testableRemoteSocketAddress says that it should only be used for testing in the javadoc. alex should add javadoc to those two methods.

i agree that we should also get a C client. alex are you willing to do that? can we do it in a separate jira?
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-ver8.patch

added a fix for ZOOKEEPER-1357
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: loadbalancing.pdf
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-12-Oct.patch

This is an updated version of patch that applies cleanly to the latest tip of trunk.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194165#comment-13194165 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12512026/ZOOKEEPER-1355-ver6.patch
  against trunk revision 1234974.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    -1 javadoc.  The javadoc tool appears to have generated 8 warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/921//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/921//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/921//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184350#comment-13184350 ] 

Alexander Shraer commented on ZOOKEEPER-1355:
---------------------------------------------

Thanks, Ben.

I posted an updated patch that addresses all your comments. I don't remember why I changed the == to >=, seems unneeded as I set currentIndex to -1 with each update (perhaps this stuck from an earlier version of the code). Anyway I removed this change.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Component/s: c client
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver12-1.patch

Changes to trunk since this patch was originally built.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183542#comment-13183542 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12510089/ZOOOKEEPER-1355.patch
  against trunk revision 1227927.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/895//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/895//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/895//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOOKEEPER-1355-test.patch
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277285#comment-13277285 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527742/ZOOKEEPER-1355-ver13.patch
  against trunk revision 1337029.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 34 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1078//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1078//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1078//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227309#comment-13227309 ] 

Marshall McMullen commented on ZOOKEEPER-1355:
----------------------------------------------

FYI, I've written the C client code to support this feature.

I've also added a bunch of tests to verify the load balancing aspect of it works correctly. One unit test is currently causing me some issues. But I intent to upload an updated patch later this evening that will include all of Alex's original changes as well as the changes to the C client.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-ver9.patch

ZOOKEEPER-1355-ver9 applies for zookeeper-3.4.3 and trunk and fixes issues raised on review-board https://reviews.apache.org/r/3781
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver13.patch

Bad patch last time, my apologies.

Bumped the version on this one to 13 to avoid confusion.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOOKEEPER-1355.patch
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183635#comment-13183635 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12510108/ZOOOKEEPER-1355-ver1.patch
  against trunk revision 1227927.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/896//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/896//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/896//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Eugene Koontz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277532#comment-13277532 ] 

Eugene Koontz commented on ZOOKEEPER-1355:
------------------------------------------

Typo in src/docs/src/documentation/content/xdocs/zookeeperProgrammers.xml: "jus" should be "just".

The documentation is good - perhaps it could make reference in the source code to where the client's server-selection logic is implemented (StaticHostProvider::updateServerList()).

i.e. provide a link to the source such as http://svn.apache.org/viewvc/zookeeper/trunk/src/java/main/org/apache/zookeeper/client/StaticHostProvider.java?view=markup or https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/client/StaticHostProvider.java

The test cases seem to have a lot more cases than the documentation: would be nice to have the doc's examples correspond to the test cases.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Patrick Hunt (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497333#comment-13497333 ] 

Patrick Hunt commented on ZOOKEEPER-1355:
-----------------------------------------

Seems reasonable to me. (cursory look). As long as it doesn't effect backward compatibility I don't see an issue.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Nov.patch, ZOOKEEPER-1355-13-Nov-ver2.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Flavio Junqueira (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246175#comment-13246175 ] 

Flavio Junqueira commented on ZOOKEEPER-1355:
---------------------------------------------

There are currently two separate reviews on the review board for this jira, one by Alex and one by Marshall. It would be great if we could keep one review request and update it with new revisions. And, just so that the comments from review board are recorded in the jira, please make sure to update the bugs field with the jira number.

I have also already reviewed the java part of this patch, but I don't see any of my previous comments reflected. 

Could you guys please fix all these? 
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-10-Oct.patch
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Mark Miller (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227283#comment-13227283 ] 

Mark Miller commented on ZOOKEEPER-1355:
----------------------------------------

Looks like this narrowly missed 3.5?
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-ver5.patch
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-13-Oct.patch

Had meant to remove the Zab Test part from this patch as Alex tells me that was already committed to trunk under another Jira.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240118#comment-13240118 ] 

Marshall McMullen commented on ZOOKEEPER-1355:
----------------------------------------------

Alex, looks like only one unit test failure -- on the Java unit test side. So it can't be related to the C changes I made.

Can you take a look when you get time?

Also, no one has reviewed my proposed C changes so would be good to start getting some eyes on it.

Note I added a new addrvec (address vector) data structure to make it easier to manage and manipulate the 3 different list of addresses (current, joining, leaving).
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227640#comment-13227640 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12518035/ZOOKEEPER-1355-ver10.patch
  against trunk revision 1297740.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/987//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/987//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/987//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491957#comment-13491957 ] 

Alexander Shraer commented on ZOOKEEPER-1355:
---------------------------------------------

Hi Flavio,

I uploaded Marshall's latest patch to review board: https://reviews.apache.org/r/3781/

Alex

                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver10.patch

Uploading ZOOKEEPER-1355-ver10.patch.

This includes all the code that was in ZOOKEEPER-1355-ver9-1.patch as well as all the C code to support this feature.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491931#comment-13491931 ] 

Marshall McMullen commented on ZOOKEEPER-1355:
----------------------------------------------

Flavio, the newest patch is not on review board as I couldn't figure out how to update review board with the latest patch... :-<
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-13-Nov.patch

Updated patch that addresses all comments in review https://reviews.apache.org/r/3781/
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Nov.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: loadbalancing-more-details.pdf

some more details added to the design document. 

Would be great if more committers comment on this patch.

thanks a lot,
alex
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217048#comment-13217048 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12516137/ZOOKEEPER-1355-ver9-1.patch
  against trunk revision 1294000.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/963//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/963//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/963//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194171#comment-13194171 ] 

Alexander Shraer commented on ZOOKEEPER-1355:
---------------------------------------------

I believe that this hits https://issues.apache.org/jira/browse/ZOOKEEPER-1357

                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver11-1.patch

Minor bug fix for count_hosts in case an empty string is passed in.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194293#comment-13194293 ] 

Patrick Hunt commented on ZOOKEEPER-1355:
-----------------------------------------

bq. I also prefer this [c client changes] to be in a separate Jira.

separate jira is fine. but I'd like to see both java and c committed at the same time.

bq. To clarify the rules some more ....

I also see you updated the docs which is great. I couldn't tell - is all the salient detail in your comments ("To clarify the rules some more ...") captured in the guide update? Great it's documented here, but in the guide is the more useful going fwd.

Thanks!

                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194313#comment-13194313 ] 

Alexander Shraer commented on ZOOKEEPER-1355:
---------------------------------------------

ok. thanks!
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver12.patch

Updated C code only to fix some unsafe locking around the server list and to properly throttle reconnect logic when we've tried all the servers in the server list unsuccessfully.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver12-2.patch

This one should apply to tip of trunk cleanly.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-ver9-1.patch

small javadoc fix. see comment for ZOOKEEPER-1355-ver9.patch
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276295#comment-13276295 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527535/ZOOKEEPER-1355-ver12-1.patch
  against trunk revision 1337029.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 34 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1074//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver12-4.patch

Updated to work correctly against tip of trunk. All unit tests should pass now.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276284#comment-13276284 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527533/ZOOKEEPER-1355-ver12.patch
  against trunk revision 1337029.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 34 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1073//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Benjamin Reed (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189553#comment-13189553 ] 

Benjamin Reed commented on ZOOKEEPER-1355:
------------------------------------------

can you freshen the patch alex?
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOOKEEPER-1355-ver1.patch
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-13-Nov-ver2.patch

Fix compile failure after refactor based on code review.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Nov.patch, ZOOKEEPER-1355-13-Nov-ver2.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-ver2.patch
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver10-1.patch

Newer version has a couple fixes:

1. Force disconnect if server set reconfig causes the server we're connected to to no longer be in list.

2. More intelligent timeout if we've tried all the servers so we don't spin.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment:     (was: ZOOKEEPER-1355-ver11-1.patch)
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13243029#comment-13243029 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12520766/ZOOKEEPER-1355-ver11-1.patch
  against trunk revision 1307644.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 34 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1025//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1025//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1025//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240984#comment-13240984 ] 

Alexander Shraer commented on ZOOKEEPER-1355:
---------------------------------------------

Thanks Marshall! the failing test seems to be the one addressed in ZOOKEEPER-1433. 
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194298#comment-13194298 ] 

Alexander Shraer commented on ZOOKEEPER-1355:
---------------------------------------------

> I also see you updated the docs which is great. I couldn't tell - is all the salient > detail in your comments ("To clarify the rules some more ...") captured in the guide > update? Great it's documented here, but in the guide is the more useful going fwd.

Yes, however the pdf in this jira is really what someone would want to read to understand the algorithm. Can we put it somewhere and reference to it from the docs ?
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194230#comment-13194230 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12512043/ZOOKEEPER-1355-ver8.patch
  against trunk revision 1234974.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/923//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/923//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/923//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227700#comment-13227700 ] 

Marshall McMullen commented on ZOOKEEPER-1355:
----------------------------------------------

I see my latest change broke existing core C tests.

I'll upload a new version.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-10-Oct.patch
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver11-1.patch

Minor bugfix in count_hosts in case an empty string is passed in.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293843#comment-13293843 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12531858/ZOOKEEPER-1355-ver14.patch
  against trunk revision 1337029.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 34 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1094//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1094//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1094//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver12-2.patch

This one should apply cleanly to tip of trunk.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277238#comment-13277238 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527735/ZOOKEEPER-1355-ver12-4.patch
  against trunk revision 1337029.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 34 new or modified tests.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1077//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191802#comment-13191802 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12511608/ZOOKEEPER-1355-ver4.patch
  against trunk revision 1234974.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    -1 findbugs.  The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/914//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/914//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/914//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver14.patch

This updated patch fixes 3 things in the C version only:

(1) Fix memleak in update_addrs in the event that the addrvec hasn't changed from last time.

(2) Fix unsafe error handling around realloc failure in addrvec.c.

(3) Fix connection failure throttling that caused it to skip a host.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227735#comment-13227735 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12518053/ZOOKEEPER-1355-ver10-4.patch
  against trunk revision 1297740.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/989//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/989//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/989//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Michi Mutsuzaki (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491679#comment-13491679 ] 

Michi Mutsuzaki commented on ZOOKEEPER-1355:
--------------------------------------------

+1 for the C client part. 

--Michi
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13497349#comment-13497349 ] 

Alexander Shraer commented on ZOOKEEPER-1355:
---------------------------------------------

I think that it doesn't affect backward compatibility.  The code works as before unless updateServerList is actually invoked. When it does, it updates the list of servers with a new shuffled list and then uses new logic to choose the first server to connect to from this list. If it succeeds to connect or tried all servers and failed to connect, we return to the "normal mode" of just going round-robin over the new shuffled list. No configuration files are touched here and nothing on the server side changed.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Nov.patch, ZOOKEEPER-1355-13-Nov-ver2.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187353#comment-13187353 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12510782/loadbalancing-more-details.pdf
  against trunk revision 1231809.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    -1 patch.  The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/908//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Benjamin Reed (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202850#comment-13202850 ] 

Benjamin Reed commented on ZOOKEEPER-1355:
------------------------------------------

mahadev, pat any more concerns or is this ready to go?
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-6-Nov-ver2.patch

Removal of srand48 seems to have broken one of the new reconfig tests. Adding it back in.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496920#comment-13496920 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12553464/ZOOKEEPER-1355-13-Nov-ver2.patch
  against trunk revision 1404288.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1261//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1261//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1261//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Nov.patch, ZOOKEEPER-1355-13-Nov-ver2.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-6-Nov.patch, ZOOKEEPER-1355-6-Nov-ver2.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184375#comment-13184375 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12510231/ZOOKEEPER%3D1355-ver3.patch
  against trunk revision 1227927.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/899//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/899//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/899//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver10-4.patch

Uploading a new version that fixes the C code to properly force a disconnect if the list of servers is modified and the host we were connected to is not in the list anymore.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183722#comment-13183722 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12510121/ZOOOKEEPER-1355-test.patch
  against trunk revision 1227927.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/897//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/897//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/897//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193591#comment-13193591 ] 

Patrick Hunt commented on ZOOKEEPER-1355:
-----------------------------------------

Good points Mahadev.

I don't see any documentation in the guides - we should provide some information there as well for users.

updateServerList is missing javadoc

also getCurrentHost (also missing javadoc) is pretty ambiguous name. In the past we haven't wanted to expose the remote server details to the client.

how is getCurrentHost different from testableRemoteSocketAddress ?

                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER=1355-ver3.patch
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Mahadev konar (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193576#comment-13193576 ] 

Mahadev konar commented on ZOOKEEPER-1355:
------------------------------------------

Ben,
 I was taking a look at it. Mind waiting till tomm? 

                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475510#comment-13475510 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12548996/ZOOKEEPER-1355-12-Oct.patch
  against trunk revision 1391526.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 34 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1221//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1221//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1221//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246415#comment-13246415 ] 

Alexander Shraer commented on ZOOKEEPER-1355:
---------------------------------------------

Done. I updated the new patch in https://reviews.apache.org/r/3781/ and answered the previous comments 
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276328#comment-13276328 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12527539/ZOOKEEPER-1355-ver12-2.patch
  against trunk revision 1337029.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 34 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1075//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1075//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1075//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Benjamin Reed (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190205#comment-13190205 ] 

Benjamin Reed commented on ZOOKEEPER-1355:
------------------------------------------

ah duh, sorry alex, that was the pdf.

ok, a couple more things:

1) can you fix the indentation of watchmanager in zookeeper.java
2) nextInReconfig needs to be private and a function comment just to explain why "InReconfig" is different
3) updateServerList needs to be thread safe. with this call there are fields: serverList, reconfig, and indices that are being accessed by the connection thread and the thread calling updateServerList.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Flavio Junqueira (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13243211#comment-13243211 ] 

Flavio Junqueira commented on ZOOKEEPER-1355:
---------------------------------------------

I'm reviewing this one.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Benjamin Reed (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benjamin Reed updated ZOOKEEPER-1355:
-------------------------------------

    Hadoop Flags: Reviewed
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-ver7.patch
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-ver6.patch

Hi, thanks for the comments!! I removed getCurrentHost, added a JavaDoc for updateServerList, and a paragraph to the documentation.

I can do the C client changes (although, after looking at the code, this scares me a bit :) ) sometime next week or the following one. 
I also prefer this to be in a separate Jira.

To clarify the rules some more, let me give you a few examples (this also appears in different levels of details in the javadoc and documentation I added to the code). 

First, in case the current host to which the client is connected is not in the new list
updateServerList will always cause the connection to be dropped. Otherwise, the decision is based on whether the number of servers has increased or decreased and by how much.

Suppose that the previous connection string contained 3 hosts and now the list contains these 3 hosts and 2 more hosts, 40% of clients connected to each of the 3 hosts will move to one of the new hosts in order to balance the load. The algorithm will cause the client 
to drop its connection to the current host to which it is connected with probability 0.4 (= 1 - 3/5, rule 1) 
and in this case cause the client to connect to one of the 2 new hosts, chosen at random. 

Another example - suppose we have 5 hosts and now update the list to remove 2 of the hosts, the clients
connected to the 3 remaining hosts will stay connected (rule 3), whereas all clients connected to the 2 removed hosts
will need to move to one of the 3 hosts, chosen at random. In this case, the formula in rule 4 simply gives 1 (3(5-3)/(3*2)) 
since no new servers were added, so clients connected to the removed hosts have no choice but connecting to the 3 old servers.
These rules also take into account the case where servers are both added and removed at the same time, and that's why rule
4 doesn't always give probability 1.

If the connection is dropped, the client moves to a special mode where he chooses a new server to connect to using the                         
probabilistic algorithm, and not jus round robin. In the first example, each client decides to disconnect from the current host with
probability 0.4 but once the decision is made, it will try to connect to a random NEW server and only if it cannot connect 
to any of the new servers will it try to connect to the old ones. 

After finding a server, or trying all servers in the new list and failing to connect, the client moves back to the normal 
mode of operation where it will pick an arbitrary server from the connectString and attempt to connect to it. 
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment:     (was: ZOOKEEPER-1355-10-Oct.patch)
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183833#comment-13183833 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12510149/ZOOKEEPER-1355-ver2.patch
  against trunk revision 1227927.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/898//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/898//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/898//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475543#comment-13475543 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12549007/ZOOKEEPER-1355-13-Oct.patch
  against trunk revision 1391526.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1222//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1222//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1222//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-12-Oct.patch, ZOOKEEPER-1355-13-Oct.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver10-4.patch

Fixing mismerge on the last patch.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473817#comment-13473817 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12548688/ZOOKEEPER-1355-10-Oct.patch
  against trunk revision 1391526.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 28 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1210//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1210//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1210//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227689#comment-13227689 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12518043/ZOOKEEPER-1355-ver10-2.patch
  against trunk revision 1297740.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/988//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/988//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/988//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Benjamin Reed (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193561#comment-13193561 ] 

Benjamin Reed commented on ZOOKEEPER-1355:
------------------------------------------

+1 i think this is ready to commit. any objections?
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Mahadev konar (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193586#comment-13193586 ] 

Mahadev konar commented on ZOOKEEPER-1355:
------------------------------------------

One more thing, what about the c client? Will we be seeing similar changes to c client? I'd very much like to keep both of them in sync if possible. We are already a little different given the security patches.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194217#comment-13194217 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12512037/ZOOKEEPER-1355-ver7.patch
  against trunk revision 1234974.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 3 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/922//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/922//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/922//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Mahadev konar (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193583#comment-13193583 ] 

Mahadev konar commented on ZOOKEEPER-1355:
------------------------------------------

Ben/Alex,
 This adds 2 public api to zookeeper handle (java). Is this intended? What the intent of getCurrentHost? 

Also, I looked at the pdf (which scares me a little - hate looking at all the math symbols :)). Can you please explain in layman terms what the process for the client to select the server to connect to? What if the server list is incorrect, what happens then? 
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Alexander Shraer (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Shraer updated ZOOKEEPER-1355:
----------------------------------------

    Attachment: ZOOKEEPER-1355-ver4.patch

ZOOKEEPER-1355-ver4.patch addresses Ben's comments. The synchronization now allows multiple threads updating the server list and another single thread trying to connect (calling next) as before.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment:     (was: ZOOKEEPER-1355-ver12-2.patch)
    
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Marshall McMullen (Updated) (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marshall McMullen updated ZOOKEEPER-1355:
-----------------------------------------

    Attachment: ZOOKEEPER-1355-ver10-2.patch

Bug fix in C code. Need to update_addrs before checking if *fd == -1 else won't seek new connection after being forcibly disconnected due to server reconfig.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOKEEPER=1355-ver3.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch, ZOOOKEEPER-1355.patch, loadbalancing-more-details.pdf, loadbalancing.pdf
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (ZOOKEEPER-1355) Add zk.updateServerList(newServerList)

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-1355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13473707#comment-13473707 ] 

Hadoop QA commented on ZOOKEEPER-1355:
--------------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12548653/ZOOKEEPER-1355-10-Oct.patch
  against trunk revision 1391526.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 31 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    -1 javac.  The patch appears to cause tar ant target to fail.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1209//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1209//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-ZOOKEEPER-Build/1209//console

This message is automatically generated.
                
> Add zk.updateServerList(newServerList) 
> ---------------------------------------
>
>                 Key: ZOOKEEPER-1355
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1355
>             Project: ZooKeeper
>          Issue Type: New Feature
>          Components: c client, java client
>            Reporter: Alexander Shraer
>            Assignee: Alexander Shraer
>             Fix For: 3.5.0
>
>         Attachments: loadbalancing-more-details.pdf, loadbalancing.pdf, ZOOKEEPER-1355-10-Oct.patch, ZOOKEEPER-1355-ver10-1.patch, ZOOKEEPER-1355-ver10-2.patch, ZOOKEEPER-1355-ver10-3.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10-4.patch, ZOOKEEPER-1355-ver10.patch, ZOOKEEPER-1355-ver11-1.patch, ZOOKEEPER-1355-ver11.patch, ZOOKEEPER-1355-ver12-1.patch, ZOOKEEPER-1355-ver12-2.patch, ZOOKEEPER-1355-ver12-4.patch, ZOOKEEPER-1355-ver12.patch, ZOOKEEPER-1355-ver13.patch, ZOOKEEPER-1355-ver14.patch, ZOOKEEPER-1355-ver2.patch, ZOOKEEPER=1355-ver3.patch, ZOOKEEPER-1355-ver4.patch, ZOOKEEPER-1355-ver5.patch, ZOOKEEPER-1355-ver6.patch, ZOOKEEPER-1355-ver7.patch, ZOOKEEPER-1355-ver8.patch, ZOOKEEPER-1355-ver9-1.patch, ZOOKEEPER-1355-ver9.patch, ZOOOKEEPER-1355.patch, ZOOOKEEPER-1355-test.patch, ZOOOKEEPER-1355-ver1.patch
>
>
> When the set of servers changes, we would like to update the server list stored by clients without restarting the clients.
> Moreover, assuming that the number of clients per server is the same (in expectation) in the old configuration (as guaranteed by the current list shuffling for example), we would like to re-balance client connections across the new set of servers in a way that a) the number of clients per server is the same for all servers (in expectation) and b) there is no excessive/unnecessary client migration.
> It is simple to achieve (a) without (b) - just re-shuffle the new list of servers at every client. But this would create unnecessary migration, which we'd like to avoid.
> We propose a simple probabilistic migration scheme that achieves (a) and (b) - each client locally decides whether and where to migrate when the list of servers changes. The attached document describes the scheme and shows an evaluation of it in Zookeeper. We also implemented re-balancing through a consistent-hashing scheme and show a comparison. We derived the probabilistic migration rules from a simple formula that we can also provide, if someone's interested in the proof.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira