[jira] [Commented] (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets

["Yun Qin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16623686#comment-16623686 ] 

Yun Qin commented on ZOOKEEPER-900:
-----------------------------------

I believe we have met an issue very similar to what's described in section b) of the original bug description. In our case, sid 1/2 weren't able to rejoin a quorum consisting of sid 3/4/5. The root cause was that QuorumCnxManager$Listener wasn't able to accept connection from sid 1/2 on sid 3/5. Restarting ZooKeeper on sid 3/5 solved the issue.

I would really appreciate it if we can proceed with the fix, now that we have a preliminary patch and people willing to take over. Thanks. 

> FLE implementation should be improved to use non-blocking sockets
> -----------------------------------------------------------------
>
>                 Key: ZOOKEEPER-900
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900
>             Project: ZooKeeper
>          Issue Type: Bug
>            Reporter: Vishal Kher
>            Assignee: Martin Kuchta
>            Priority: Critical
>             Fix For: 3.6.0, 3.5.5
>
>         Attachments: ZOOKEEPER-900-part2.patch, ZOOKEEPER-900.patch, ZOOKEEPER-900.patch1, ZOOKEEPER-900.patch2
>
>
> From earlier email exchanges:
> 1. Blocking connects and accepts:
> a) The first problem is in manager.toSend(). This invokes connectOne(), which does a blocking connect. While testing, I changed the code so that connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run() does a socketChannel.connect(). After starting AsyncConnect, connectOne starts a timer. connectOne continues with normal operations if the connection is established before the timer expires, otherwise, when the timer expires it interrupts AsyncConnect() thread and returns. In this way, I can have an upper bound on the amount of time we need to wait for connect to succeed. Of course, this was a quick fix for my testing. Ideally, we should use Selector to do non-blocking connects/accepts. I am planning to do that later once we at least have a quick fix for the problem and consensus from others for the real fix (this problem is big blocker for us). Note that it is OK to do blocking IO in SenderWorker and RecvWorker threads since they block IO to the respective peer.
> b) The blocking IO problem is not just restricted to connectOne(), but also in receiveConnection(). The Listener thread calls receiveConnection() for each incoming connection request. receiveConnection does blocking IO to get peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the peer that had sent the connection request. All of this is happening from the Listener. In short, if a peer fails after initiating a connection, the Listener thread won't be able to accept connections from other peers, because it would be stuck in read() or connetOne(). Also the code has an inherent cycle. initiateConnection() and receiveConnection() will have to be very carefully synchronized otherwise, we could run into deadlocks. This code is going to be difficult to maintain/modify.
> Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)