[jira] [Created] (HBASE-9393) Hbase dose not closing a closed socket resulting in many CLOSE_WAIT

["Avi Zrachya (JIRA)" <ji...@apache.org>]

Avi Zrachya created HBASE-9393:
----------------------------------

             Summary: Hbase dose not closing a closed socket resulting in many CLOSE_WAIT 
                 Key: HBASE-9393
                 URL: https://issues.apache.org/jira/browse/HBASE-9393
             Project: HBase
          Issue Type: Bug
    Affects Versions: 0.94.2
         Environment: Centos 6.4 - 7 regionservers/datanodes, 8 TB per node, 7279 regions
            Reporter: Avi Zrachya


HBase dose not close a dead connection with the datanode.
This resulting in over 60K CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port.

The example below is with low CLOSE_WAIT count because we had to restart hbase to solve the porblem, later in time it will incease to 60-100K sockets on CLOSE_WAIT

[root@hd2-region3 ~]# netstat -nap |grep CLOSE_WAIT |grep 21592 |wc -l
13156
[root@hd2-region3 ~]# ps -ef |grep 21592
root     17255 17219  0 12:26 pts/0    00:00:00 grep 21592
hbase    21592     1 17 Aug29 ?        03:29:06 /usr/java/jdk1.6.0_26/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx8000m -ea -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-regionserver-hd2-region3.swnet.corp.log ...




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira