You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Duo Zhang (JIRA)" <ji...@apache.org> on 2019/06/07 12:58:00 UTC
[jira] [Commented] (HBASE-22477) Throwing exception when meta
region is not in OPEN state in client registry may crash a master
[ https://issues.apache.org/jira/browse/HBASE-22477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16858603#comment-16858603 ]
Duo Zhang commented on HBASE-22477:
-----------------------------------
OK, another failure introduced by this behavior...
https://builds.apache.org/job/HBase-Flaky-Tests/job/master/3428/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.TestReplicationDroppedTables-output.txt/*view*/
{noformat}
2019-06-06 18:54:31,361 ERROR [master/asf902:0:becomeActiveMaster] helpers.MarkerIgnoringBase(159): Failed to become active master
org.apache.hadoop.hbase.client.RetriesExhaustedException: Cannot get the location for replica0 of region for in hbase:meta
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:335)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:263)
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:405)
at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:285)
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:564)
at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:766)
at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734)
at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:690)
at org.apache.hadoop.hbase.MetaTableAccessor.fullScanRegions(MetaTableAccessor.java:220)
at org.apache.hadoop.hbase.master.assignment.RegionStateStore.visitMeta(RegionStateStore.java:77)
at org.apache.hadoop.hbase.master.assignment.AssignmentManager.loadMeta(AssignmentManager.java:1294)
at org.apache.hadoop.hbase.master.assignment.AssignmentManager.joinCluster(AssignmentManager.java:1255)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1100)
at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2376)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:605)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Meta region is in state OPENING
at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$1(ZKAsyncRegistry.java:162)
at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:81)
at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)
at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)
... 1 more
{noformat}
Let's get this in?
[~zghaobac] Thanks.
> Throwing exception when meta region is not in OPEN state in client registry may crash a master
> ----------------------------------------------------------------------------------------------
>
> Key: HBASE-22477
> URL: https://issues.apache.org/jira/browse/HBASE-22477
> Project: HBase
> Issue Type: Bug
> Components: Client, master, meta
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Priority: Major
> Attachments: HBASE-22477-test.patch
>
>
> {noformat}
> 2019-05-26 17:10:31,195 ERROR [master/asf906:0:becomeActiveMaster] helpers.MarkerIgnoringBase(159): Failed to become active master
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Cannot get the location for replica0 of region for in hbase:meta
> at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:335)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
> at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:263)
> at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:405)
> at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:285)
> at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:564)
> at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:766)
> at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734)
> at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:690)
> at org.apache.hadoop.hbase.MetaTableAccessor.fullScanRegions(MetaTableAccessor.java:220)
> at org.apache.hadoop.hbase.master.assignment.RegionStateStore.visitMeta(RegionStateStore.java:77)
> at org.apache.hadoop.hbase.master.assignment.AssignmentManager.loadMeta(AssignmentManager.java:1294)
> at org.apache.hadoop.hbase.master.assignment.AssignmentManager.joinCluster(AssignmentManager.java:1255)
> at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1100)
> at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2375)
> at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:605)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Meta region is in state OPENING
> at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$1(ZKAsyncRegistry.java:162)
> at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:70)
> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:81)
> at org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:70)
> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)
> at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)
> ... 1 more
> {noformat}
> I think the logic is introduced at the time that we do not change the state of meta region on zk. But now, we will change the state of meta region, so maybe we should remove the logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)