[jira] [Created] (HADOOP-15946) the Connection thread should notify all calls in finally clause before quit.

["Jinglun (JIRA)" <ji...@apache.org>]

Jinglun created HADOOP-15946:
--------------------------------

             Summary: the Connection thread should notify all calls in finally clause before quit.
                 Key: HADOOP-15946
                 URL: https://issues.apache.org/jira/browse/HADOOP-15946
             Project: Hadoop Common
          Issue Type: Improvement
            Reporter: Jinglun
         Attachments: issue-replay.patch

Threads that call Client.call() would wait forever unless the connection thread notifies them, so the connection thread should try it's best to notify when it's going to quit.

In Connection.close(), if any Throwable occurs before cleanupCalls(), the connection thread will quit directly and leave all the waiting threads waiting forever. So i think doing cleanupCalls() in finally clause might be a good idea.

I met this problem when i started a hadoop2.6 DataNode with 8 block pools. The DN successfully reported to 7 Namespaces and failed at the last Namespace because the connection thread of the heartbeat rpc got a "OOME:Direct buffer memory" and quit without calling cleanupCalls().

I think we can move cleanupCalls() to finally clause as a protection, because i notice in HADOOP-10940 the close of stream is changed to IOUtils.closeStream(ipcStreams) which catches all Throwable, so the problem i met was fixed. 

issue-replay.patch simulates the case i described above.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org