You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@zookeeper.apache.org by GitBox <gi...@apache.org> on 2021/01/18 20:23:47 UTC

[GitHub] [zookeeper] functioner opened a new pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

functioner opened a new pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582


   I provide a fix for the issue [ZOOKEEPER-4074](https://issues.apache.org/jira/browse/ZOOKEEPER-4074) that I proposed.
   I basically by default enable learner.asyncSending which is introduced in [ZOOKEEPER-3575](https://issues.apache.org/jira/browse/ZOOKEEPER-3575), which is already tested. I also update the document for the learner.asyncSending configuration.
   I modify a test case in LearnerTest a little bit because enabling learner.asyncSending may require more initialization in Learner for testing.
   ZOOKEEPER-3575 will not take effect until 3.7.0, so I am also wondering if we need to backport this patch to 3.6.x?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner commented on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner commented on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-773550072






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner closed pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner closed pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner closed pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner closed pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner commented on a change in pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner commented on a change in pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#discussion_r582016045



##########
File path: zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
##########
@@ -1149,6 +1149,11 @@ property, when available, is noted below.
     **New in 3.6.2:**
     When enabled, a learner will close the quorum socket asynchronously. This is useful for TLS connections where closing a socket might take a long time, block the shutdown process, potentially delay a new leader election, and leave the quorum unavailabe. Closing the socket asynchronously avoids blocking the shutdown process despite the long socket closing time and a new leader election can be started while the socket being closed. The default is false.
 
+* *learner.asyncSending*
+    (Jave system property only: **learner.closeSocketAsync**)

Review comment:
       Oh, yeah, it's a typo. I'll fix it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner edited a comment on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner edited a comment on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-785154916


   @TisonKun  To add a test for this issue, we may consider adding a fault injection tool, e.g., Byteman ([ZOOKEEPER-3601](https://issues.apache.org/jira/browse/ZOOKEEPER-3601) by @maoling ). I have provided the script to reproduce this bug ([ZOOKEEPER-4074](https://issues.apache.org/jira/browse/ZOOKEEPER-4074)) using Byteman.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner commented on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner commented on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-800714986


   > I am not sure whether we should enable this functionality by default in branch-3.6.
   > This functionality is new and it has changed core ZK code, I think it is better not to enable it by default to avoid impact on other functionalities.
   > But I am sure we should backport to branch-3.6.
   > 
   > @functioner what you say shall we keep disabled it by default ?
   
   The asynchronous sending feature has been introduced to the codebase for a while with proper testing. So my thinking is that it might be mainly out of temporary development purposes that the property was not enabled by default. If that's the case, it might be useful to keep it enabled by default now because this feature makes the system more resilient to network failures. Users may not change default parameters (especially given that this is a Java system property), so they may encounter the failures; and it may take a while for them to realize that surprisingly the system actually has a resilient feature that could have prevented that failure. So I'm inclined to advocate changing the default to be on. But I understand your concern that in general changing default parameter should be done carefully.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] arshadmohammad commented on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
arshadmohammad commented on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-800813149


   > Users may not change default parameters (especially given that this is a Java system property),
   
   There is no reason to keep the configuration only a java system property. It should be configurable through zoo.cfg. As the changes are not released yet we can change and make it configurable through zoo.cfg . I will create a bug
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner commented on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner commented on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-785154916


   @TisonKun  To add a test for this issue, we may consider adding a fault injection tool, e.g., Byteman ([ZOOKEEPER-3601](https://issues.apache.org/jira/browse/ZOOKEEPER-3601) by @maoling ).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] maoling commented on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
maoling commented on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-774423810


   @functioner  The CI failure may not your blame. We're looking at this patch. Thanks for your contribution.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] TisonKun commented on a change in pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
TisonKun commented on a change in pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#discussion_r582004181



##########
File path: zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md
##########
@@ -1149,6 +1149,11 @@ property, when available, is noted below.
     **New in 3.6.2:**
     When enabled, a learner will close the quorum socket asynchronously. This is useful for TLS connections where closing a socket might take a long time, block the shutdown process, potentially delay a new leader election, and leave the quorum unavailabe. Closing the socket asynchronously avoids blocking the shutdown process despite the long socket closing time and a new leader election can be started while the socket being closed. The default is false.
 
+* *learner.asyncSending*
+    (Jave system property only: **learner.closeSocketAsync**)

Review comment:
       typo?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner edited a comment on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner edited a comment on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-785154916


   @TisonKun  To add a test for this issue, we may consider adding a fault injection tool, e.g., Byteman ([ZOOKEEPER-3601](https://issues.apache.org/jira/browse/ZOOKEEPER-3601) by @maoling ). I have provided the script to reproduce this bug using Byteman.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner commented on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner commented on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-773553110


   @maoling I've fixed the failing test cases in my local machine where all test cases pass. But there are still some failing test cases in the CI server, and I was wondering what could be the reason? Do you have any suggestion?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] functioner commented on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
functioner commented on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-773550072


   > I provide a fix for the issue [ZOOKEEPER-4074](https://issues.apache.org/jira/browse/ZOOKEEPER-4074) that I proposed.
   > I basically by default enable learner.asyncSending introduced by [ZOOKEEPER-3575](https://issues.apache.org/jira/browse/ZOOKEEPER-3575), which is already tested. I also update the document for the learner.asyncSending configuration.
   > I modify a test case in LearnerTest a little bit because enabling learner.asyncSending may require more initialization in Learner for testing.
   > ZOOKEEPER-3575 will not take effect until 3.7.0, so I am also wondering if we need to backport this patch to 3.6.x?
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [zookeeper] arshadmohammad commented on pull request #1582: ZOOKEEPER-4074: Network issue while Learner is executing writePacket can cause the follower to hang

Posted by GitBox <gi...@apache.org>.
arshadmohammad commented on pull request #1582:
URL: https://github.com/apache/zookeeper/pull/1582#issuecomment-797269112


   I am not sure whether we should enable this functionality by default in branch-3.6.
   This functionality is new and it has changed core ZK code, I think it is better not to enable it by default to avoid impact on other functionalities.
   But I am sure we should backport to branch-3.6.
   
   @functioner what you say shall we keep disabled it by default ?
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org