You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2019/02/01 20:01:01 UTC
[jira] [Updated] (HBASE-19937) Ensure createRSGroupTable be called
after ProcedureExecutor and LoadBalancer are initialized
[ https://issues.apache.org/jira/browse/HBASE-19937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-19937:
-----------------------------------
Fix Version/s: (was: 1.5.0)
> Ensure createRSGroupTable be called after ProcedureExecutor and LoadBalancer are initialized
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-19937
> URL: https://issues.apache.org/jira/browse/HBASE-19937
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
> Affects Versions: 2.0.0-beta-2
> Reporter: Xiaolin Ha
> Assignee: Xiaolin Ha
> Priority: Major
> Fix For: 2.0.0-beta-2, 1.4.2, 2.0.0
>
> Attachments: HBASE-19937.branch-2.001.patch, HBASE-19937.branch-2.002.patch, HBASE-19937.branch-2.003.patch, HBASE-19937.branch-2.004.patch, HBASE-19937.branch-2.005.patch, HBASE-19937.branch-2.006.patch, import-order.png
>
>
> hbase:rsgroup table will be created by calling createRSGroupTable when master load system coprocessors in
>
> {code:java}
> 844 this.cpHost = new MasterCoprocessorHost(this, this.conf);{code}
> when ProcedureExecutor hasn't been initialized before createRSGroupTable, it will encounter Exception as follows,
>
> {noformat}
> Exception in thread "org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker-localhost,49715,1518088607130" java.lang.IllegalArgumentException
> at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:120)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:847)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:835)
> at org.apache.hadoop.hbase.master.HMaster.createSystemTable(HMaster.java:1795)
> at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:858)
> at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.waitForGroupTableOnline(RSGroupInfoManagerImpl.java:823)
> at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.run(RSGroupInfoManagerImpl.java:743){noformat}
> while ProcedureExecutor initialized by calling
> {code:java}
> 848 startServiceThreads();{code}
> And LoadBalancer is initialized by calling
> {code:java}
> 868 this.balancer.initialize();{code}
> When LoadBalancer hasn't been initialized before createRSGroupTable, it will encounters Exception as follows,
> 2018-02-02,16:12:45,688 ERROR org.apache.hadoop.hbase.procedure2.ProcedureExecutor: CODE-BUG: Uncaught runtime exception: pid=7, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS; CreateTableProcedure table=hbase:rsgroup
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.generateGroupMaps(RSGroupBasedLoadBalancer.java:254)
> at org.apache.hadoop.hbase.rsgroup.RSGroupBasedLoadBalancer.roundRobinAssignment(RSGroupBasedLoadBalancer.java:162)
> at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createRoundRobinAssignProcedures(AssignmentManager.java:603)
> at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:108)
> at org.apache.hadoop.hbase.master.procedure.CreateTableProcedure.executeFromState(CreateTableProcedure.java:51)
> at org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:182)
> at org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:845)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1458)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1227)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:78)
> at org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1738)
>
> As a result of CreateTableProcedure.rollbackState, it may then print logs warning TableExistsException as follows,
> 2018-02-02,16:12:55,503 WARN org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: Failed to perform check
> java.io.IOException: Failed to create group table. org.apache.hadoop.hbase.TableExistsException: hbase:rsgroup
> at org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker.createRSGroupTable(RSGroupInfoManagerImpl.java:877)
>
> After some auto-retries, it loops running the thread RSGroupStartupWorker, will print logs as follows,
> 2018-02-02,16:23:17,626 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
> 2018-02-02,16:23:17,730 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
> 2018-02-02,16:23:17,834 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
> 2018-02-02,16:23:17,937 INFO org.apache.hadoop.hbase.rsgroup.RSGroupInfoManagerImpl$RSGroupStartupWorker: RSGroup table=hbase:rsgroup isOnline=true, regionCount=0, assignCount=0, rootMetaFound=true
>
> And using shells of rsgroup, it will tips that currently is in "offline mode".
>
> The reason of this problem is that the order of createRSGroupTable and initializing of ProcedureExecutor and LoadBalancer is out of control. If the former is excuted earlier, it will encounter Exception mentioned before.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)