You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@zookeeper.apache.org by GitBox <gi...@apache.org> on 2020/03/23 11:44:21 UTC
[GitHub] [zookeeper] symat opened a new pull request #1293: ZOOKEEPER-3756:
Members slow to rejoin quorum using Kubernetes
symat opened a new pull request #1293: ZOOKEEPER-3756: Members slow to rejoin quorum using Kubernetes
URL: https://github.com/apache/zookeeper/pull/1293
Whenever we close the current master ZooKeeper server, a new leader election
is triggered. During the new election, a connection will be established between
all the servers, by calling the synchronized 'connectOne' method in
QuorumCnxManager. The method will open the socket and send a single small
initial message to the other server, usually very quickly. If the destination
host is unreachable, it should fail immediately.
However, when we use Kubernetes, then the destination host is always reachable
as it points to Kubernetes services. If the actual container / pod is not
available then the 'socket.connect' method will timeout (by default after 5 sec)
instead of failing immediately with NoRouteToHostException. As the 'connectOne'
method is synchronized, this timeout will block the creation of other
connections, so a single unreachable host can cause timeout in the leader
election protocol.
One workaround is to decrease the socket connection timeout with the
'-Dzookeeper.cnxTimeout' stystem property, but the proper fix would be to
make the connection initiation fully asynchronous, as using very low timeout can
have its own side effect. Fortunately most of the initial message sending
is already made async: the SASL authentication can take more time, so the
second (authentication + initial message sending) part of the initiation protocol
is already called in a separate thread, when Quorum SASL authentication is enabled.
In the following patch I made the whole connection initiation async, by
always using the async executor (not only when Quorum SASL is enabled) and
also moving the socket.connect call into the async thread.
I also created a unit test to verify my fix. I added a static socket factory that can be
changed by the tests using a packet private setter method. My test failed (and
produced the same error logs as we see in the original Jira ticket) before I applied
my changes and a time-outed as no leader election succeeded after 15 seconds.
After the changes the test runs very quickly, in 1-2 seconds.
Note: due to the multiAddress changes, we will need different PRs to the branch 3.5
and to the 3.6+ branches. I will submit the other PR once this got reviewed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zookeeper] nkalmar commented on issue #1293: ZOOKEEPER-3756:
Members slow to rejoin quorum using Kubernetes
Posted by GitBox <gi...@apache.org>.
nkalmar commented on issue #1293: ZOOKEEPER-3756: Members slow to rejoin quorum using Kubernetes
URL: https://github.com/apache/zookeeper/pull/1293#issuecomment-604532914
Merged to branch-3.5, thanks @symat
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zookeeper] symat commented on issue #1293: ZOOKEEPER-3756: Members
slow to rejoin quorum using Kubernetes
Posted by GitBox <gi...@apache.org>.
symat commented on issue #1293: ZOOKEEPER-3756: Members slow to rejoin quorum using Kubernetes
URL: https://github.com/apache/zookeeper/pull/1293#issuecomment-604277486
@eolivelli can you please take a look?
this is the same as #1289 just for branch 3.5...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zookeeper] symat commented on issue #1293: ZOOKEEPER-3756: Members
slow to rejoin quorum using Kubernetes
Posted by GitBox <gi...@apache.org>.
symat commented on issue #1293: ZOOKEEPER-3756: Members slow to rejoin quorum using Kubernetes
URL: https://github.com/apache/zookeeper/pull/1293#issuecomment-602543525
FYI: the same fix can be found for 3.6+ here:
https://github.com/apache/zookeeper/pull/1289
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zookeeper] nkalmar closed pull request #1293: ZOOKEEPER-3756:
Members slow to rejoin quorum using Kubernetes
Posted by GitBox <gi...@apache.org>.
nkalmar closed pull request #1293: ZOOKEEPER-3756: Members slow to rejoin quorum using Kubernetes
URL: https://github.com/apache/zookeeper/pull/1293
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services