You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by iv...@apache.org on 2017/12/14 22:31:27 UTC

[bookkeeper] 02/02: Race in PCBC connection if channel inactive before callback

This is an automated email from the ASF dual-hosted git repository.

ivank pushed a commit to branch branch-4.6
in repository https://gitbox.apache.org/repos/asf/bookkeeper.git

commit 9c7079a82ee64520e1056652216a445a7531668f
Author: Ivan Kelly <iv...@apache.org>
AuthorDate: Mon Dec 11 21:57:51 2017 +0100

    Race in PCBC connection if channel inactive before callback
    
    There's a race in PCBC where if the channel is errored before the
    connection callback occurs, the channel will hang around as a zombie
    forever.
    
    This caused a flake in
    TestDisableEnsembleChange#testRetryFailureBookie, but the problem
    itself could affect production code. This test hit it because it tries
    to reconnect in a tight loop as a bookie is booting.
    
    The error occurs because we only change state with channelInactive if
    the channelInactive occurred on the channel that matches the currently
    assigned this.channel. If the channel connects successfully, but
    channelInactive occurs before the connect callback runs, the channel
    will be closed, but the state will be transitioned to CONNECTED. No
    more events will occur on the channel, so it'll hang around forever,
    and all requests on the channel will timeout.
    
    The fix is to check that the channel is active before assigning it to
    this.channel and setting state to CONNECTED. If the channel is not
    active at this point, then the else clause of this block will run
    which closes the channel, and sets state to DISCONNECTED. In this
    case, the next request will trigger a new connection attempt.
    
    Author: Ivan Kelly <iv...@apache.org>
    
    Reviewers: Matteo Merli <mm...@apache.org>
    
    This closes #827 from ivankelly/ensemble-flake
---
 .../main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java   | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
index c392c79..3d9ba13 100644
--- a/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
+++ b/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/PerChannelBookieClient.java
@@ -1851,7 +1851,7 @@ public class PerChannelBookieClient extends ChannelInboundHandlerAdapter {
             Queue<GenericCallback<PerChannelBookieClient>> oldPendingOps;
 
             synchronized (PerChannelBookieClient.this) {
-                if (future.isSuccess() && state == ConnectionState.CONNECTING) {
+                if (future.isSuccess() && state == ConnectionState.CONNECTING && future.channel().isActive()) {
                     LOG.info("Successfully connected to bookie: {}", future.channel());
                     rc = BKException.Code.OK;
                     channel = future.channel();

-- 
To stop receiving notification emails like this one, please contact
"commits@bookkeeper.apache.org" <co...@bookkeeper.apache.org>.