[jira] [Comment Edited] (HBASE-16488) Starting namespace and quota services in master startup asynchronizely

["Allan Yang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-16488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067623#comment-16067623 ] 

Allan Yang edited comment on HBASE-16488 at 6/29/17 2:57 AM:
-------------------------------------------------------------

why do we need to wait here? shouldn't we throw a exception fast to tell the caller the namespace manager is not ready? Otherwise, many calls like createnamespace/modifynamespace from user will suffer from a strange timeout.

{code}
 void checkNamespaceManagerReady() throws IOException {
     checkInitialized();
-    if (tableNamespaceManager == null ||
-        !tableNamespaceManager.isTableAvailableAndInitialized(true)) {
+    if (tableNamespaceManager == null) {
       throw new IOException("Table Namespace Manager not ready yet, try again later");
+    } else if (!tableNamespaceManager.isTableAvailableAndInitialized(true)) {
+      try {
+        // Wait some time.
+        long startTime = EnvironmentEdgeManager.currentTime();
+        int timeout = conf.getInt("hbase.master.namespace.wait.for.ready", 30000);
+        while (!tableNamespaceManager.isTableNamespaceManagerStarted() &&
+            EnvironmentEdgeManager.currentTime() - startTime < timeout) {
+          Thread.sleep(100);
+        }
+      } catch (InterruptedException e) {
+        throw (InterruptedIOException) new InterruptedIOException().initCause(e);
+      }
+      if (!tableNamespaceManager.isTableNamespaceManagerStarted()) {
+        throw new IOException("Table Namespace Manager not fully initialized, try again later");
+      }
     }
   }
{code}


was (Author: allan163):
why do we need to wait here? shouldn't we throw a exception fast to tell the caller the namespace manager is not ready? Otherwise, many calls like createnamespace/modifynamespace from will suffer from a strange timeout.

{code}
 void checkNamespaceManagerReady() throws IOException {
     checkInitialized();
-    if (tableNamespaceManager == null ||
-        !tableNamespaceManager.isTableAvailableAndInitialized(true)) {
+    if (tableNamespaceManager == null) {
       throw new IOException("Table Namespace Manager not ready yet, try again later");
+    } else if (!tableNamespaceManager.isTableAvailableAndInitialized(true)) {
+      try {
+        // Wait some time.
+        long startTime = EnvironmentEdgeManager.currentTime();
+        int timeout = conf.getInt("hbase.master.namespace.wait.for.ready", 30000);
+        while (!tableNamespaceManager.isTableNamespaceManagerStarted() &&
+            EnvironmentEdgeManager.currentTime() - startTime < timeout) {
+          Thread.sleep(100);
+        }
+      } catch (InterruptedException e) {
+        throw (InterruptedIOException) new InterruptedIOException().initCause(e);
+      }
+      if (!tableNamespaceManager.isTableNamespaceManagerStarted()) {
+        throw new IOException("Table Namespace Manager not fully initialized, try again later");
+      }
     }
   }
{code}

> Starting namespace and quota services in master startup asynchronizely
> ----------------------------------------------------------------------
>
>                 Key: HBASE-16488
>                 URL: https://issues.apache.org/jira/browse/HBASE-16488
>             Project: HBase
>          Issue Type: Improvement
>          Components: master
>    Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>            Reporter: Stephen Yuan Jiang
>            Assignee: Stephen Yuan Jiang
>         Attachments: HBASE-16488.v1-branch-1.patch, HBASE-16488.v1-master.patch, HBASE-16488.v2-branch-1.patch, HBASE-16488.v2-branch-1.patch, HBASE-16488.v3-branch-1.patch, HBASE-16488.v3-branch-1.patch, HBASE-16488.v4-branch-1.patch, HBASE-16488.v5-branch-1.patch, HBASE-16488.v6-branch-1.patch, HBASE-16488.v7-branch-1.patch, HBASE-16488.v8-branch-1.patch
>
>
> From time to time, during internal IT test and from customer, we often see master initialization failed due to namespace table region takes long time to assign (eg. sometimes split log takes long time or hanging; or sometimes RS is temporarily not available; sometimes due to some unknown assignment issue).  In the past, there was some proposal to improve this situation, eg. HBASE-13556 / HBASE-14190 (Assign system tables ahead of user region assignment) or HBASE-13557 (Special WAL handling for system tables) or  HBASE-14623 (Implement dedicated WAL for system tables).  
> This JIRA proposes another way to solve this master initialization fail issue: namespace service is only used by a handful operations (eg. create table / namespace DDL / get namespace API / some RS group DDL).  Only quota manager depends on it and quota management is off by default.  Therefore, namespace service is not really needed for master to be functional.  So we could start namespace service asynchronizely without blocking master startup.
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)