Socket Reader Thread was crashed with Hadoop Writable engine, and client reached to the dead end

[Anfernee Xu <an...@gmail.com>]

Hi experts,

I'm running with Hadoop 2.2.0,  I have a distributed application written
with Hadoop Writable and I'm pretty happy with it, pretty stable. But a few
days back, I ran into one strange issue, one client was stucking in talking
to one of the servers, the RPC was made in a background thread and I have a
timer to guard against timeout, the call was timeout as expected, then
another user made the same call to the same server, still timeout, after
some time, I found I ran out of threads for the threadpool which is used
for making the RPC call. Also I found the Hadoop RPC Socket Reader thread
was crashed for some reason(I cannot find it from JConsole) on the server
side. I also looked the code, I think below code is problematic.

org.apache.hadoop.ipc.Client.call method, around line#1336, as below

while (!call.done) {
try
{
     call.wait(); // wait for the result }
catch (InterruptedException ie)
{ // save the fact that we were interrupted

 interrupted = true; }

}

Looks like even I tried to interrupt the call, the client was still trying
to wait for the response, is this a bug?

Thanks

-- 
--Anfernee