Hbase does not closing a closed socket resulting in thousand of CLOSE_WAIT sockets

[徐景辉 <xu...@neusoft.com>]

Guys:

I got the fatal error: HBase dose not close a dead connection with the
datanode. And I have reviewed the codes of Hadoop 2.4.0/HBase 0.98.0. I
found the patched codes of
<https://issues.apache.org/jira/browse/HBASE-9393> HBASE-9393 and
<https://issues.apache.org/jira/browse/HDFS-5671> HDFS-5671 have added into
the release of Hadoop 2.4.0/HBase 0.98.0. These codes are in the end of
method getRemoteBlockReaderFromTcp() in class BlockReaderFactory:

      } finally {

        if (blockReader == null) {

          IOUtils.cleanup(LOG, peer);

        }

      }

So I guess a new bug lead my problem and I create a new issue
<https://issues.apache.org/jira/browse/HBASE-11833> HBASE-11833. Do you guys
got the same issue? If that, please let me know. Thanks a lot.

 

Problem Description:
This resulting in over 30K+ CLOSE_WAIT and at some point HBase can not
connect to the datanode because too many mapped sockets from one host to
another on the same port:50010. 
After I restart all RSs, the count of CLOSE_WAIT will increase always.
$ netstat -an|grep CLOSE_WAIT|wc -l
2545

1.    netstat -nap|grep CLOSE_WAIT|grep 6569|wc -l
2545

2.    ps -ef|grep 6569
hbase 6569 6556 21 Aug25 ? 09:52:33 /opt/jdk1.6.0_25/bin/java
-Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m
-XX:+UseConcMarkSweepGC

I aslo have reviewed these issues:
 <https://issues.apache.org/jira/browse/HBASE-9393> HBASE-9393
 <https://issues.apache.org/jira/browse/HDFS-5671> HDFS-5671
 <https://issues.apache.org/jira/browse/HDFS-1836> HDFS-1836



 

 

---------------------------------------------------------------------------------------------------
Confidentiality Notice: The information contained in this e-mail and any accompanying attachment(s) 
is intended only for the use of the intended recipient and may be confidential and/or privileged of 
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of this communication is 
not the intended recipient, unauthorized use, forwarding, printing,  storing, disclosure or copying 
is strictly prohibited, and may be unlawful.If you have received this communication in error,please 
immediately notify the sender by return e-mail, and delete the original message and all copies from 
your system. Thank you. 
---------------------------------------------------------------------------------------------------