You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "kangkang.guo (Jira)" <ji...@apache.org> on 2021/06/03 03:38:00 UTC
[jira] [Assigned] (HBASE-25954) When graceful_stop a regionserver,
RegionMover always retry to move the regions to other rsgoup's
regionservers
[ https://issues.apache.org/jira/browse/HBASE-25954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
kangkang.guo reassigned HBASE-25954:
------------------------------------
Assignee: kangkang.guo
> When graceful_stop a regionserver, RegionMover always retry to move the regions to other rsgoup's regionservers
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-25954
> URL: https://issues.apache.org/jira/browse/HBASE-25954
> Project: HBase
> Issue Type: Bug
> Reporter: Zezhen Jia
> Assignee: kangkang.guo
> Priority: Critical
>
> [hadoop@bigdata-hbase-master003.prod.com bin]$ ./graceful_stop.sh bigdata-hbase-rs105.prod.com 2021-05-28T11:05:06 Disabling load balancer SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/cloud/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop/cloud/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2021-05-28T11:05:12 Previous balancer state was true 2021-05-28T11:05:12 Unloading bigdata-hbase-rs105.prod.com region(s) SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/cloud/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop/cloud/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 2021-05-28 11:05:13,990 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:host.name=bigdata-hbase-master003.prod.com 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_241 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:java.home=/home/hadoop/cloud/jdk1.8.0_241/jre 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: adoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2-tests.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.9.2.jar:/home/hadoop/cloud/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9.2.jar:/home/hadoop/cloud/hadoop/contrib/capacity-scheduler/*.jar:/home/hadoop/cloud/hbase/lib/client-facing-thirdparty/slf4j-log4j12-1.7.25.jar 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:java.library.path=/home/hadoop/cloud/hadoop/lib/native 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:os.name=Linux 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-696.3.1.el6.x86_64 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:user.name=hadoop 2021-05-28 11:05:13,991 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop 2021-05-28 11:05:13,992 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop/cloud/hbase/bin 2021-05-28 11:05:13,993 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Initiating client connection, connectString=bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181 sessionTimeout=30000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$14/668288113@5d84a9ed 2021-05-28 11:05:14,138 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)] zookeeper.Login: Client successfully logged in. 2021-05-28 11:05:14,140 INFO [Thread-4] zookeeper.Login: TGT refresh thread started. 2021-05-28 11:05:14,142 INFO [Thread-4] zookeeper.Login: TGT valid starting at: Fri May 28 11:05:14 CST 2021 2021-05-28 11:05:14,142 INFO [Thread-4] zookeeper.Login: TGT expires: Fri May 28 23:05:14 CST 2021 2021-05-28 11:05:14,143 INFO [Thread-4] zookeeper.Login: TGT refresh sleeping until: Fri May 28 21:07:54 CST 2021 2021-05-28 11:05:14,144 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. 2021-05-28 11:05:14,150 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server bigdata-hbase-rs101.prod.com/10.87.103.134:2181. Will attempt to SASL-authenticate using Login Context section 'Client' 2021-05-28 11:05:14,154 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)] zookeeper.ClientCnxn: Socket connection established to bigdata-hbase-rs101.prod.com/10.87.103.134:2181, initiating session 2021-05-28 11:05:14,158 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-SendThread(bigdata-hbase-rs101.prod.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server bigdata-hbase-rs101.prod.com/10.87.103.134:2181, sessionid = 0x371ba929aec16a2, negotiated timeout = 30000 2021-05-28 11:05:14,754 INFO [pool-2-thread-1] util.RegionMover: Moving 16 regions from bigdata-hbase-rs105.prod.com to 4 servers using 1 threads .Ack Mode:true 2021-05-28 11:05:20,479 INFO [pool-3-thread-1] util.RegionMover: Moving region:0e75b85e4970faf3ee7d83ba5edfbf7a from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs101.prod.com,16020,1615279448828
> 2021-05-28 11:06:14,775 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28 11:06:14,786 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c] zookeeper.ZooKeeper: Session: 0x371ba929aec16a2 closed 2021-05-28 11:06:14,786 INFO [ReadOnlyZKClient-bigdata-hbase-master003.prod.com:2181,bigdata-hbase-master004.prod.com:2181,bigdata-hbase-rs101.prod.com:2181,bigdata-hbase-rs102.prod.com:2181,bigdata-hbase-rs103.prod.com:2181@0x24aed80c-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x371ba929aec16a2 2021-05-28 11:06:18,880 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5 2021-05-28 11:06:19,983 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:06:21,086 INFO [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28 11:06:22,190 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,080, !1r^,1609991498659.0e75b85e4970faf3ee7d83ba5edfbf7a. stuck on bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs101.prod.com,16020,1615279448828 2021-05-28 11:06:22,210 INFO [pool-3-thread-1] util.RegionMover: Moving region:cad8bd3f09cb02b6be55acfa3fe81509 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs102.prod.com,16020,1615279437177
> 2021-05-28 11:07:22,265 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28 11:07:23,369 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5 2021-05-28 11:07:24,472 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:07:25,576 INFO [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28 11:07:26,679 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,0c0, 4B.J,1609991498659.cad8bd3f09cb02b6be55acfa3fe81509. stuck on bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs102.prod.com,16020,1615279437177 2021-05-28 11:07:26,699 INFO [pool-3-thread-1] util.RegionMover: Moving region:175b9149fe5c2715832fe8ebe2273e3e from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs103.prod.com,16020,1615279437032 2021-05-28 11:07:27,868 INFO [pool-3-thread-1] util.RegionMover: Moved Region test:t2,100, "la,1608747109269.175b9149fe5c2715832fe8ebe2273e3e. cost:1.188 2021-05-28 11:07:27,890 INFO [pool-3-thread-1] util.RegionMover: Moving region:8e58792c240f9dd6879fa610de648978 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs104.prod.com,16020,1615279437314 2021-05-28 11:07:29,042 INFO [pool-3-thread-1] util.RegionMover: Moved Region test:t2,180, &C9#,1608747109269.8e58792c240f9dd6879fa610de648978. cost:1.174 2021-05-28 11:07:29,061 INFO [pool-3-thread-1] util.RegionMover: Moving region:b135ab7358a5375eabec1ba9bd6abea2 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs101.prod.com,16020,1615279448828 2021-05-28 11:08:29,144 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28 11:08:30,248 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5 2021-05-28 11:08:31,351 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:08:32,454 INFO [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28 11:08:33,556 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,280,101866,1608761306479.b135ab7358a5375eabec1ba9bd6abea2. stuck on bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs101.prod.com,16020,1615279448828 2021-05-28 11:08:33,576 INFO [pool-3-thread-1] util.RegionMover: Moving region:9ab60065e4d739b4b1205baaeb4ab112 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs102.prod.com,16020,1615279437177 2021-05-28 11:09:33,626 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28 11:09:34,729 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5 2021-05-28 11:09:35,833 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:09:36,936 INFO [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28 11:09:38,038 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,380,7408,1608742520235.9ab60065e4d739b4b1205baaeb4ab112. stuck on bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs102.prod.com,16020,1615279437177 2021-05-28 11:09:38,052 INFO [pool-3-thread-1] util.RegionMover: Moving region:36d483f9537cbcb066c535d5c97500b7 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs103.prod.com,16020,1615279437032
> 2021-05-28 11:10:38,056 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28 11:10:39,159 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5 2021-05-28 11:10:40,261 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:10:41,364 INFO [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28 11:10:42,467 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,480,7627,1608755031336.36d483f9537cbcb066c535d5c97500b7. stuck on bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs103.prod.com,16020,1615279437032 2021-05-28 11:10:42,491 INFO [pool-3-thread-1] util.RegionMover: Moving region:4fb21d1bcdd1230351828b7c6e779e08 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs104.prod.com,16020,1615279437314 2021-05-28 11:10:43,629 INFO [pool-3-thread-1] util.RegionMover: Moved Region test:t2,60000000,1608752443610.4fb21d1bcdd1230351828b7c6e779e08. cost:1.161 2021-05-28 11:10:43,659 INFO [pool-3-thread-1] util.RegionMover: Moving region:3286929984e362752b03dd4f1c723b78 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs101.prod.com,16020,1615279448828
> 2021-05-28 11:11:43,652 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28 11:11:44,755 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5 2021-05-28 11:11:45,858 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:11:46,961 INFO [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28 11:11:48,063 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,700,73323,1608755402890.3286929984e362752b03dd4f1c723b78. stuck on bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs101.prod.com,16020,1615279448828 2021-05-28 11:11:48,089 INFO [pool-3-thread-1] util.RegionMover: Moving region:1626df981007111128002bb7008ebe07 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs102.prod.com,16020,1615279437177 2021-05-28 11:11:49,209 INFO [pool-3-thread-1] util.RegionMover: Moved Region test:t2,80000000,1608747946269.1626df981007111128002bb7008ebe07. cost:1.146 2021-05-28 11:11:49,234 INFO [pool-3-thread-1] util.RegionMover: Moving region:b9934989ffd9abb8b2484208b2a4860c from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs103.prod.com,16020,1615279437032 2021-05-28 11:12:49,215 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28 11:12:50,318 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5 2021-05-28 11:12:51,421 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:12:52,523 INFO [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28 11:12:53,626 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,900,7405,1608752559891.b9934989ffd9abb8b2484208b2a4860c. stuck on bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs103.prod.com,16020,1615279437032 2021-05-28 11:12:53,657 INFO [pool-3-thread-1] util.RegionMover: Moving region:2cfb6d72b252e02b6651741bce8af0a9 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs104.prod.com,16020,1615279437314 2021-05-28 11:12:54,789 INFO [pool-3-thread-1] util.RegionMover: Moved Region test:t2,980,5003,1608752559891.2cfb6d72b252e02b6651741bce8af0a9. cost:1.163 2021-05-28 11:12:54,805 INFO [pool-3-thread-1] util.RegionMover: Moving region:027a58629de39ed42fd534364f239bc1 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs101.prod.com,16020,1615279448828 2021-05-28 11:12:55,918 INFO [pool-3-thread-1] util.RegionMover: Moved Region test:t2,a0000000,1608748813503.027a58629de39ed42fd534364f239bc1. cost:1.129 2021-05-28 11:12:55,941 INFO [pool-3-thread-1] util.RegionMover: Moving region:2f40e2ea776ad5ccc63dedc112f780b3 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs102.prod.com,16020,1615279437177
> 2021-05-28 11:13:55,918 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28 11:13:57,022 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5 2021-05-28 11:13:58,124 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:13:59,227 INFO [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 ^[2021-05-28 11:14:00,330 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,c0000000,1608749249432.2f40e2ea776ad5ccc63dedc112f780b3. stuck on bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs102.prod.com,16020,1615279437177 2021-05-28 11:14:00,376 INFO [pool-3-thread-1] util.RegionMover: Moving region:bebf648e765b7369ff2a86c6ca290828 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs103.prod.com,16020,1615279437032 2021-05-28 11:15:00,409 INFO [pool-3-thread-1] util.RegionMover: Retry 1 of maximum 5 2021-05-28 11:15:01,512 INFO [pool-3-thread-1] util.RegionMover: Retry 2 of maximum 5 2021-05-28 11:15:02,615 INFO [pool-3-thread-1] util.RegionMover: Retry 3 of maximum 5 2021-05-28 11:15:03,718 INFO [pool-3-thread-1] util.RegionMover: Retry 4 of maximum 5 2021-05-28 11:15:04,821 ERROR [pool-3-thread-1] util.RegionMover: Region: test:t2,e80, _R@_,1608746950666.bebf648e765b7369ff2a86c6ca290828. stuck on bigdata-hbase-rs105.prod.com,16020,1616402972076,newServer=bigdata-hbase-rs103.prod.com,16020,1615279437032 2021-05-28 11:15:04,844 INFO [pool-3-thread-1] util.RegionMover: Moving region:5beda27c11cef510a70af7f4609c84d3 from bigdata-hbase-rs105.prod.com,16020,1616402972076 to bigdata-hbase-rs104.prod.com,16020,1615279437314 2021-05-28 11:15:05,994 INFO [pool-3-thread-1] util.RegionMover: Moved Region test:t2,f80, !<&2,1608745215226.5beda27c11cef510a70af7f4609c84d3. cost:1.172 2021-05-28 11:15:05,994 ERROR [pool-2-thread-1] util.RegionMover: Was Not able to move region....Exiting Now 2021-05-28 11:15:05,995 ERROR [pool-2-thread-1] util.RegionMover: Error while unloading regions java.lang.Exception: Could not move region Exception at org.apache.hadoop.hbase.util.RegionMover.waitMoveTasksToFinish(RegionMover.java:541) at org.apache.hadoop.hbase.util.RegionMover.unloadRegions(RegionMover.java:497) at org.apache.hadoop.hbase.util.RegionMover.lambda$unload$1(RegionMover.java:449) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-05-28 11:15:06,003 INFO [main] client.ConnectionImplementation: Closing master protocol: MasterService 2021-05-28T11:15:06 Unloaded bigdata-hbase-rs105.prod.com region(s) 2021-05-28T11:15:06 Stopping regionserver on bigdata-hbase-rs105.prod.com bigdata-hbase-rs105.prod.com: running regionserver, logging to /home/logs/hbase/hbase-hadoop-regionserver-bigdata-hbase-rs105.prod.com.out bigdata-hbase-rs105.prod.com: stopping regionserver.... 2021-05-28T11:15:10 Restoring balancer state to true
--
This message was sent by Atlassian Jira
(v8.3.4#803005)