[jira] [Commented] (ZOOKEEPER-2251) Add Client side packet response timeout to avoid infinite wait.

["Akihiro Suda (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706468#comment-14706468 ] 

Akihiro Suda commented on ZOOKEEPER-2251:
-----------------------------------------

Hello, I'm trying to reproduce this bug.

However I'm not sure about the meaning of "only few packets missed".
If some IP packets were lost, TCP layer (in the OS kernel) should be able to detect and retransmit lost packets.
So I guess the lost packets are eventually delivered to the client OS, but ZK client get hangs due to unexpected delay of the delivered packets.

Could you please share your client workload and its thread dump?

BTW, for partial packet loss simulation, [Linux netem|http://www.linuxfoundation.org/collaborate/workgroups/networking/netem#Packet_loss] or my [Earthquake tool|http://osrg.github.io/earthquake/post/zookeeper-2212/](found ZOOKEEPER-2212, and perhaps also reproduced ZOOKEEPER-2080) might be useful. 
(Wireshark is a packet analyzer, and doesn't provide any network simulation feature.)


> Add Client side packet response timeout to avoid infinite wait.
> ---------------------------------------------------------------
>
>                 Key: ZOOKEEPER-2251
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2251
>             Project: ZooKeeper
>          Issue Type: Bug
>            Reporter: nijel
>
> I came across one issue related to Client side packet response timeout In my cluster many packet drops happened for some time.
> One observation is the zookeeper client got hanged. As per the thread dump it is waiting for the response/ACK for the operation performed (synchronous API used here).
> I am using zookeeper.serverCnxnFactory=org.apache.zookeeper.server.NIOServerCnxnFactory
> Since only few packets missed there is no DISCONNECTED event occurred.
> Need add a "response time out" for the operations or packets.
> *Comments from [~rakeshr]*
> My observation about the problem:-
> * Can use tools like 'Wireshark' to simulate the artificial packet loss.
> * Assume there is only one packet in the 'outgoingQueue' and unfortunately the server response packet lost. Now, client will enter into infinite waiting. https://github.com/apache/zookeeper/blob/trunk/src/java/main/org/apache/zookeeper/ClientCnxn.java#L1515
> * Probably we can discuss more about this problem and possible solutions(add packet ACK timeout or another better approach) in the jira.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)