You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "steven xu (JIRA)" <ji...@apache.org> on 2014/08/27 05:06:57 UTC

[jira] [Updated] (HBASE-11833) Hbase does not closing a closed socket resulting in thousand of CLOSE_WAIT sockets

     [ https://issues.apache.org/jira/browse/HBASE-11833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

steven xu updated HBASE-11833:
------------------------------

    Description: 
HBase dose not close a dead connection with the datanode.
This resulting in over 30K+ CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port:50010. 
After I restart all RSs,  the count of CLOSE_WAIT will increase always.
$ netstat -an|grep CLOSE_WAIT|wc -l
2545
# netstat -nap|grep CLOSE_WAIT|grep 6569|wc -l
2545
# ps -ef|grep 6569
hbase     6569  6556 21 Aug25 ?        09:52:33 /opt/jdk1.6.0_25/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC 

I aslo have reviewed these issues:
[HBASE-9393|https://issues.apache.org/jira/browse/HBASE-9393]
[HDFS-5671|https://issues.apache.org/jira/browse/HDFS-5671]
[HDFS-1836|https://issues.apache.org/jira/browse/HDFS-1836]
I found HBase 0.98/Hadoop 2.4.0 I uesed which source codes are not different from these patches.
But I donot understand why HBase 0.98/Hadoop 2.4.0 also have this isssue. Please check. Thanks a lot.

  was:
HBase dose not close a dead connection with the datanode.
This resulting in over 30K+ CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port:50010. 
After I restart all RSs,  the count of CLOSE_WAIT will increase always.
$ netstat -an|grep CLOSE_WAIT|wc -l
2545
# netstat -nap|grep CLOSE_WAIT|grep 6569|wc -l
2545
# ps -ef|grep 6569
hbase     6569  6556 21 Aug25 ?        09:52:33 /opt/jdk1.6.0_25/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC 

I aslo have reviewed these issues:
[HBASE-9393|https://issues.apache.org/jira/browse/HBASE-9393]
[HDFS-5671|https://issues.apache.org/jira/browse/HDFS-5671]
[HDFS-1836|https://issues.apache.org/jira/browse/HDFS-1836]
I found HBase 0.98/Hadoop 2.4.0 I uesed which source codes are different from these patches.
But I donot understand why HBase 0.98/Hadoop 2.4.0 also have this isssue. Please check. Thanks a lot.


> Hbase does not closing a closed socket resulting in thousand of CLOSE_WAIT sockets
> ----------------------------------------------------------------------------------
>
>                 Key: HBASE-11833
>                 URL: https://issues.apache.org/jira/browse/HBASE-11833
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.98.0
>         Environment: RHEL 6.3 -HDP 2.1 -6 RegionServers/Datanode -18T per node -3108Regions
>            Reporter: steven xu
>
> HBase dose not close a dead connection with the datanode.
> This resulting in over 30K+ CLOSE_WAIT and at some point HBase can not connect to the datanode because too many mapped sockets from one host to another on the same port:50010. 
> After I restart all RSs,  the count of CLOSE_WAIT will increase always.
> $ netstat -an|grep CLOSE_WAIT|wc -l
> 2545
> # netstat -nap|grep CLOSE_WAIT|grep 6569|wc -l
> 2545
> # ps -ef|grep 6569
> hbase     6569  6556 21 Aug25 ?        09:52:33 /opt/jdk1.6.0_25/bin/java -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p -Xmx1000m -XX:+UseConcMarkSweepGC 
> I aslo have reviewed these issues:
> [HBASE-9393|https://issues.apache.org/jira/browse/HBASE-9393]
> [HDFS-5671|https://issues.apache.org/jira/browse/HDFS-5671]
> [HDFS-1836|https://issues.apache.org/jira/browse/HDFS-1836]
> I found HBase 0.98/Hadoop 2.4.0 I uesed which source codes are not different from these patches.
> But I donot understand why HBase 0.98/Hadoop 2.4.0 also have this isssue. Please check. Thanks a lot.



--
This message was sent by Atlassian JIRA
(v6.2#6252)