You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ZhongYou Li (Jira)" <ji...@apache.org> on 2024/06/16 08:20:00 UTC
[jira] [Updated] (HBASE-28669) After one RegionServer restarts, another RegionServer leaks a connection to ZooKeeper
[ https://issues.apache.org/jira/browse/HBASE-28669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ZhongYou Li updated HBASE-28669:
--------------------------------
Description:
The peer "to_pd_A" has been removed, but there is an error log in RegionServer, error log:
{code:java}
2024-06-11 09:42:34.074 ERROR [ReplicationExecutor-0.replicationSource,to_pd_A-172.30.112.11,6002,1709612684705-SendThread(bjtx-hbase-onll-meta-01:2181)] client.StaticHostProvider: Unable to resolve address: bjtx-hbase-onll-meta-03:2181
java.net.UnknownHostException: bjtx-hbase-onll-meta-03
at java.net.InetAddress$CachedAddresses.get(InetAddress.java:764)
at java.net.InetAddress.getAllByName0(InetAddress.java:1291)
at java.net.InetAddress.getAllByName(InetAddress.java:1144)
at java.net.InetAddress.getAllByName(InetAddress.java:1065)
at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137){code}
Here are the steps to reproduce:
I have 3 RegionServers. The following steps can reproduce the phenomenon of ZK connection leakage:
1. Turn on the Replication function
2. Create a peer
3. Shut down any two RegionServers for a few minutes and restart them
4. Print the thread stack on the RegionServer that is not shut down, search for the keyword <peerId>, and you can see that there are 4 more threads with ZooKeeper
5. By removing the peer, the extra 4 threads still exist
The following is the thread stack leak in one of my RegionServers:
{code:java}
"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.29,6002,1718180442225-EventThread" #610 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=466.94s tid=0x00007efc58179000 nid=0x5a051 waiting on condition [0x00007efc2cdef000]
"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.29,6002,1718180442225-SendThread(10.0.16.100:2181)" #609 daemon prio=5 os_prio=0 cpu=3.02ms elapsed=466.94s tid=0x00007efc58178800 nid=0x5a050 runnable [0x00007efc2cef0000]
"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.9,6002,1718180457260-EventThread" #505 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=556.09s tid=0x00007efc50094800 nid=0x59c04 waiting on condition [0x00007efc2d7f7000]
"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.9,6002,1718180457260-SendThread(10.0.16.100:2181)" #504 daemon prio=5 os_prio=0 cpu=3.72ms elapsed=556.09s tid=0x00007efc50093000 nid=0x59c03 runnable [0x00007efc2d8f8000] {code}
was:
Original error log:
{code:java}
2024-06-11 09:42:34.074 ERROR [ReplicationExecutor-0.replicationSource,to_pd_A-172.30.112.11,6002,1709612684705-SendThread(bjtx-hbase-onll-meta-01:2181)] client.StaticHostProvider: Unable to resolve address: bjtx-hbase-onll-meta-03:2181
java.net.UnknownHostException: bjtx-hbase-onll-meta-03
at java.net.InetAddress$CachedAddresses.get(InetAddress.java:764)
at java.net.InetAddress.getAllByName0(InetAddress.java:1291)
at java.net.InetAddress.getAllByName(InetAddress.java:1144)
at java.net.InetAddress.getAllByName(InetAddress.java:1065)
at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137){code}
The peer "to_pd_A" has been removed.
Here are the steps to reproduce:
I have 3 RegionServers. The following steps can reproduce the phenomenon of ZK connection leakage:
1. Turn on the Replication function
2. Create a peer
3. Shut down any two RegionServers for a few minutes and restart them
4. Print the thread stack on the RegionServer that is not shut down, search for the keyword <peerId>, and you can see that there are 4 more threads with ZooKeeper
5. By removing the peer, the extra 4 threads still exist
The following is the thread stack leak in one of my RegionServers:
{code:java}
"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.29,6002,1718180442225-EventThread" #610 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=466.94s tid=0x00007efc58179000 nid=0x5a051 waiting on condition [0x00007efc2cdef000]
"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.29,6002,1718180442225-SendThread(10.0.16.100:2181)" #609 daemon prio=5 os_prio=0 cpu=3.02ms elapsed=466.94s tid=0x00007efc58178800 nid=0x5a050 runnable [0x00007efc2cef0000]
"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.9,6002,1718180457260-EventThread" #505 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=556.09s tid=0x00007efc50094800 nid=0x59c04 waiting on condition [0x00007efc2d7f7000]
"ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.9,6002,1718180457260-SendThread(10.0.16.100:2181)" #504 daemon prio=5 os_prio=0 cpu=3.72ms elapsed=556.09s tid=0x00007efc50093000 nid=0x59c03 runnable [0x00007efc2d8f8000] {code}
> After one RegionServer restarts, another RegionServer leaks a connection to ZooKeeper
> -------------------------------------------------------------------------------------
>
> Key: HBASE-28669
> URL: https://issues.apache.org/jira/browse/HBASE-28669
> Project: HBase
> Issue Type: Bug
> Components: Replication
> Affects Versions: 2.4.5
> Reporter: ZhongYou Li
> Priority: Minor
> Labels: Replication
>
> The peer "to_pd_A" has been removed, but there is an error log in RegionServer, error log:
> {code:java}
> 2024-06-11 09:42:34.074 ERROR [ReplicationExecutor-0.replicationSource,to_pd_A-172.30.112.11,6002,1709612684705-SendThread(bjtx-hbase-onll-meta-01:2181)] client.StaticHostProvider: Unable to resolve address: bjtx-hbase-onll-meta-03:2181
> java.net.UnknownHostException: bjtx-hbase-onll-meta-03
> at java.net.InetAddress$CachedAddresses.get(InetAddress.java:764)
> at java.net.InetAddress.getAllByName0(InetAddress.java:1291)
> at java.net.InetAddress.getAllByName(InetAddress.java:1144)
> at java.net.InetAddress.getAllByName(InetAddress.java:1065)
> at org.apache.zookeeper.client.StaticHostProvider$1.getAllByName(StaticHostProvider.java:92)
> at org.apache.zookeeper.client.StaticHostProvider.resolve(StaticHostProvider.java:147)
> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:375)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137){code}
> Here are the steps to reproduce:
> I have 3 RegionServers. The following steps can reproduce the phenomenon of ZK connection leakage:
> 1. Turn on the Replication function
> 2. Create a peer
> 3. Shut down any two RegionServers for a few minutes and restart them
> 4. Print the thread stack on the RegionServer that is not shut down, search for the keyword <peerId>, and you can see that there are 4 more threads with ZooKeeper
> 5. By removing the peer, the extra 4 threads still exist
>
> The following is the thread stack leak in one of my RegionServers:
> {code:java}
> "ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.29,6002,1718180442225-EventThread" #610 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=466.94s tid=0x00007efc58179000 nid=0x5a051 waiting on condition [0x00007efc2cdef000]
> "ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.29,6002,1718180442225-SendThread(10.0.16.100:2181)" #609 daemon prio=5 os_prio=0 cpu=3.02ms elapsed=466.94s tid=0x00007efc58178800 nid=0x5a050 runnable [0x00007efc2cef0000]
> "ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.9,6002,1718180457260-EventThread" #505 daemon prio=5 os_prio=0 cpu=0.27ms elapsed=556.09s tid=0x00007efc50094800 nid=0x59c04 waiting on condition [0x00007efc2d7f7000]
> "ReplicationExecutor-0.replicationSource,lizy_test_replication-10.0.16.9,6002,1718180457260-SendThread(10.0.16.100:2181)" #504 daemon prio=5 os_prio=0 cpu=3.72ms elapsed=556.09s tid=0x00007efc50093000 nid=0x59c03 runnable [0x00007efc2d8f8000] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)