[jira] [Updated] (HADOOP-6762) exception while doing RPC I/O closes channel

["Todd Lipcon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HADOOP-6762:
--------------------------------

    Attachment: hadoop-6762.txt

Here's an updated patch against trunk.

I ran all of the unit tests in the ipc package locally and they passed. I also tried the new unit tests _without_ the patch, and they failed as expected.

Given that there was a deadlock found in an early rev of this patch, I also ran all of the IPC unit tests under jcarder to look for lock inversions and it found none.

I ran the RPCCallBenchmark for 30 seconds with and without the patch, with the following results:

With patch:
====== Results ======
Options:
rpcEngine=class org.apache.hadoop.ipc.ProtobufRpcEngine
serverThreads=30
serverReaderThreads=4
clientThreads=30
host=0.0.0.0
port=12345
secondsToRun=30
msgSize=1024
Total calls per second: 24668.0
CPU time per call on client: 58639 ns
CPU time per call on server: 64893 ns


Without patch:
====== Results ======
Options:
rpcEngine=class org.apache.hadoop.ipc.ProtobufRpcEngine
serverThreads=30
serverReaderThreads=4
clientThreads=30
host=0.0.0.0
port=12345
secondsToRun=30
msgSize=1024
Total calls per second: 27881.0
CPU time per call on client: 68079 ns
CPU time per call on server: 62582 ns

As expected, the CPU time on the client was increased and the throughput went down by about 13%, since the RPC calls are now being shuttled between threads on the client side. That's unfortunate, but given that this fixes an important bug, and given that _client_ side RPC throughput is rarely a bottleneck in common usage scenarios, I think it is acceptable.

This patch is also nearly identical to a patch that we've shipped in CDH since June 2010, so I'm fairly confident that the approach is correct.
                
> exception while doing RPC I/O closes channel
> --------------------------------------------
>
>                 Key: HADOOP-6762
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6762
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 0.20.2
>            Reporter: sam rash
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hadoop-6762-10.txt, hadoop-6762-1.txt, hadoop-6762-2.txt, hadoop-6762-3.txt, hadoop-6762-4.txt, hadoop-6762-6.txt, hadoop-6762-7.txt, hadoop-6762-8.txt, hadoop-6762-9.txt, HADOOP-6762.patch, hadoop-6762.txt, hadoop-6762.txt, hadoop-6762.txt
>
>
> If a single process creates two unique fileSystems to the same NN using FileSystem.newInstance(), and one of them issues a close(), the leasechecker thread is interrupted.  This interrupt races with the rpc namenode.renew() and can cause a ClosedByInterruptException.  This closes the underlying channel and the other filesystem, sharing the connection will get errors.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira