[jira] [Comment Edited] (HBASE-14190) Assign system tables ahead of user region assignment

["Allan Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-14190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067704#comment-16067704 ] 

Allan Yang edited comment on HBASE-14190 at 6/29/17 4:17 AM:
-------------------------------------------------------------

We are sometimes facing master take a long time to be inited because of namespace table not assigned either. This issue and HBASE-16488 is solving this problem in branch-1. To my understand, the namespace table is the only system table blocking master from startup. Since this issue and HBASE-16488 have both provide a complex patch and not committed for a long time. Here I offer a simple but (maybe)hack solution to discuss:
In {{openRegion}} of RSRpcServices, we have one thread only for OpenMetaHandler to open meta region, namespace table region and other regions are opened in a shared thread pool(So namespace region may be queued in the pool because other region is opening or blocked when open). since
1. the one thread for  OpenMetaHandler is never used after meta have been assigned
2. namespace region always assign after meta region is assigned when cluster restart
 Why not use this thread for namespace table region opening?
The changes are simple, something like this:
{code}
 // If there is no action in progress, we can submit a specific handler.
          // Need to pass the expected version in the constructor.
          if (region.isMetaRegion() || region.isNamespaceTable()) { //assign namespace table using OpenMetaHandler
            regionServer.service.submit(new OpenMetaHandler(
              regionServer, regionServer, region, htd, masterSystemTime, coordination, ord));
          } else {
....
}
{code}
Ping [~ted_yu],[~syuanjiang],[~stack]


was (Author: allan163):
We are sometimes facing master take a long time to be inited because of namespace table not assigned either. This issue and HBASE-16488 is solving this problem in branch-1. To my understand, the namespace table is the only system table blocking master from startup. Since this issue and HBASE-16488 have both provide a complex patch and not committed for a long time. Here I offer a simple but (maybe)hack solution to discuss:
In {{openRegion}} of RSRpcServices, we have one thread only for OpenMetaHandler to open meta region, namespace table region and other regions are opened in a shared thread pool. since
1. the one thread for  OpenMetaHandler is never used after meta have been assigned
2. namespace region always assign after meta region is assigned when cluster restart
 Why not use this thread for namespace table region opening?
The changes are simple, something like this:
{code}
 // If there is no action in progress, we can submit a specific handler.
          // Need to pass the expected version in the constructor.
          if (region.isMetaRegion() || region.isNamespaceTable()) { //assign namespace table using OpenMetaHandler
            regionServer.service.submit(new OpenMetaHandler(
              regionServer, regionServer, region, htd, masterSystemTime, coordination, ord));
          } else {
....
}
{code}
Ping [~ted_yu],[~syuanjiang],[~stack]

> Assign system tables ahead of user region assignment
> ----------------------------------------------------
>
>                 Key: HBASE-14190
>                 URL: https://issues.apache.org/jira/browse/HBASE-14190
>             Project: HBase
>          Issue Type: Bug
>          Components: Region Assignment
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Critical
>         Attachments: 14190-system-wal-v1.txt, 14190-v12.4.txt, 14190-v12.txt
>
>
> Currently the namespace table region is assigned like user regions.
> I spent several hours working with a customer where master couldn't finish initialization.
> Even though master was restarted quite a few times, it went down with the following:
> {code}
> 2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: []
> 2015-08-05 17:16:57,530 FATAL [hdpmaster1:60000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned
>   at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
>   at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:985)
>   at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:779)
>   at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:182)
>   at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1646)
>   at java.lang.Thread.run(Thread.java:744)
> {code}
> During previous run(s), namespace table was created, hence leaving an entry in hbase:meta.
> The following if block in TableNamespaceManager#start() was skipped:
> {code}
>     if (!MetaTableAccessor.tableExists(masterServices.getConnection(),
>       TableName.NAMESPACE_TABLE_NAME)) {
> {code}
> TableNamespaceManager#start() spins, waiting for namespace region to be assigned.
> There was issue in master assigning user regions.
> We tried issuing 'assign' command from hbase shell which didn't work because of the following check in MasterRpcServices#assignRegion():
> {code}
>       master.checkInitialized();
> {code}
> This scenario can be avoided if we assign hbase:namespace table after hbase:meta is assigned but before user table region assignment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)