You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/10/27 23:25:44 UTC

[jira] Created: (HBASE-964) Startup stuck "waiting for root region"

Startup stuck "waiting for root region"
---------------------------------------

                 Key: HBASE-964
                 URL: https://issues.apache.org/jira/browse/HBASE-964
             Project: Hadoop HBase
          Issue Type: Bug
         Environment: TRUNK
            Reporter: stack
            Assignee: Jim Kellerman


Seems easy to reproduce.

I am trying to start a master then start a regionserver.  I'm using local filesystem.

I start master like this:

{code}
./bin/hbase master start
{code}

The regionserver similarly.

All comes up. Then I try to run sequentialWriter from PE.

Here is what shows in master log:

{code}
...
08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
...
{code}


Here is the regionserver log:

{code}
....
08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000

...
{code}

and so on.

Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HBASE-964) Startup stuck "waiting for root region"

Posted by "stack (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-964:
------------------------

    Fix Version/s: 0.19.0

Adding to 0.19.0.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman resolved HBASE-964.
---------------------------------

    Resolution: Fixed

The latest patch seems to have resolved the problem.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.19.0
>
>         Attachments: 964-logs-1.zip, 964-v2.patch, 964.patch, hbase-964-v1.patch
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman resolved HBASE-964.
---------------------------------

    Resolution: Fixed

Works now. Committed.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman reassigned HBASE-964:
-----------------------------------

    Assignee: Jim Kellerman  (was: stack)

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.19.0
>
>         Attachments: 964-logs-1.zip, 964-v2.patch, 964.patch, hbase-964-v1.patch
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-964) Startup stuck "waiting for root region"

Posted by "stack (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645282#action_12645282 ] 

stack commented on HBASE-964:
-----------------------------

I see the issue in your workspace too -- its just missing the tell-tale Sleeping messages:
{code}
2008-11-05 00:30:15,604 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225845013136
2008-11-05 00:30:15,604 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225845013136
2008-11-05 00:30:55,670 ERROR org.apache.hadoop.hbase.RegionHistorian: Unable to create RegionHistorian
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:793)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:439)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:421)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:449)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:414)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:113)
        at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:310)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:970)
        at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:901)
        at java.lang.Thread.run(Thread.java:619)
{code}

We're stuck for 40 seconds trying to open region because we can't locate root region though its been successfully deployed a good few minutes earlier (Something to do w/ all running on one regionserver?).

Take the issue back if you want to try and figure it yourself.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (HBASE-964) Startup stuck "waiting for root region"

Posted by "stack (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack reassigned HBASE-964:
---------------------------

    Assignee: stack  (was: Jim Kellerman)

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HBASE-964:
--------------------------------

    Attachment: 964.patch

This patch works for me and does not generate the errors:

ERROR org.apache.hadoop.hbase.RegionHistorian: Unable to create RegionHistorian
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region

and it restores the strictness of HMaster.findRootRegion which is necessary for 'safe-mode'

Please verify that this patch works for you.

Thanks.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.19.0
>
>         Attachments: 964-logs-1.zip, 964.patch, hbase-964-v1.patch
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Purtell updated HBASE-964:
---------------------------------

    Attachment: 964-logs-1.zip

I am also seeing this:

ERROR org.apache.hadoop.hbase.RegionHistorian: Unable to create RegionHistorian
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:793)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:439)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:421)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:449)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:414)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:113)
        at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:310)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:970)
        at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:901)
        at java.lang.Thread.run(Thread.java:619)

I have a very small test master/regionserver setup running latest trunk where I can reliably reproduce this every time on start up. DEBUG level logs are attached. The logs are large because I waited until all regions (29) finally came up.

I can test any patches you'd like. 

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.19.0
>
>         Attachments: 964-logs-1.zip
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645117#action_12645117 ] 

Jim Kellerman commented on HBASE-964:
-------------------------------------

Cannot reproduce with hbase-trunk-latest and either hadoop 0.19 rc0 or hadoop 0.19 branch latest

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-964) Startup stuck "waiting for root region"

Posted by "stack (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645132#action_12645132 ] 

stack commented on HBASE-964:
-----------------------------

I see this exception all over the place -- in eclipse and up in cluster.  I'll take this issue and try and fix it.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645131#action_12645131 ] 

Jim Kellerman commented on HBASE-964:
-------------------------------------

Just the PE table. It should not matter as other tables do not get deployed until the meta does because they aren't found until the meta is online.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-964) Startup stuck "waiting for root region"

Posted by "stack (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645130#action_12645130 ] 

stack commented on HBASE-964:
-----------------------------

Any tables in your instance?

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-964) Startup stuck "waiting for root region"

Posted by "stack (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644916#action_12644916 ] 

stack commented on HBASE-964:
-----------------------------

Are we sure this is right?

Testing, I still see this everytime I startup:

{code}
2008-11-04 08:01:54,113 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
2008-11-04 08:02:39,320 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225785758336
2008-11-04 08:02:39,320 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225785758336
2008-11-04 08:02:39,334 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:41,336 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:43,339 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:45,341 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:47,344 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:49,347 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:51,348 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:53,351 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:55,353 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:57,356 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:02:59,358 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
2008-11-04 08:02:59,359 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:01,361 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:03,362 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:05,365 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:07,367 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:09,369 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:11,372 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:13,374 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:15,375 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:17,378 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
2008-11-04 08:03:19,381 ERROR org.apache.hadoop.hbase.RegionHistorian: Unable to create RegionHistorian
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:793)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:439)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:421)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:449)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:414)
        at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:113)
        at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:310)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:970)
        at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:901)
        at java.lang.Thread.run(Thread.java:619)
{code}

Then it starts to work.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Reopened: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman reopened HBASE-964:
---------------------------------


Stack still sees this issue, but I cannot reproduce it. Since it is still a problem, reopening issue.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.19.0
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman updated HBASE-964:
--------------------------------

    Attachment: 964-v2.patch

This patch 'latches' safe-mode so that when a split happens we don't run a foul of allRegionsAssigned.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.19.0
>
>         Attachments: 964-logs-1.zip, 964-v2.patch, 964.patch, hbase-964-v1.patch
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.

     [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jean-Daniel Cryans updated HBASE-964:
-------------------------------------

    Attachment: hbase-964-v1.patch

I just committed this patch to trunk. I lessens the requirements to get to root region from the master because it was too restrictive. Like in HBASE-919, RegionManager.allRegionsAssigned() was preventing something to happen and here it was happening because HRS wanted to initialize the RegionHistorian and it has to do it during the first non-catalog region opening. But because HRS wasn't able to report it to the master, the region is considered as unassigned and the HTable instantiation inside RegionHistorian couldn't happen. I also removed the sleep in HBaseClusterTest.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.19.0
>
>         Attachments: 964-logs-1.zip, hbase-964-v1.patch
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HBASE-964) Startup stuck "waiting for root region"

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.

    [ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12646316#action_12646316 ] 

Jean-Daniel Cryans commented on HBASE-964:
------------------------------------------

+1 on latest patch.

> Startup stuck "waiting for root region"
> ---------------------------------------
>
>                 Key: HBASE-964
>                 URL: https://issues.apache.org/jira/browse/HBASE-964
>             Project: Hadoop HBase
>          Issue Type: Bug
>         Environment: TRUNK
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.19.0
>
>         Attachments: 964-logs-1.zip, 964-v2.patch, 964.patch, hbase-964-v1.patch
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver.  I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.