You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2008/11/05 19:57:44 UTC
[jira] Updated: (HBASE-964) Startup stuck "waiting for root region"
[ https://issues.apache.org/jira/browse/HBASE-964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-964:
---------------------------------
Attachment: 964-logs-1.zip
I am also seeing this:
ERROR org.apache.hadoop.hbase.RegionHistorian: Unable to create RegionHistorian
org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:793)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:439)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:421)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:449)
at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:414)
at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:113)
at org.apache.hadoop.hbase.RegionHistorian.online(RegionHistorian.java:310)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:970)
at org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:901)
at java.lang.Thread.run(Thread.java:619)
I have a very small test master/regionserver setup running latest trunk where I can reliably reproduce this every time on start up. DEBUG level logs are attached. The logs are large because I waited until all regions (29) finally came up.
I can test any patches you'd like.
> Startup stuck "waiting for root region"
> ---------------------------------------
>
> Key: HBASE-964
> URL: https://issues.apache.org/jira/browse/HBASE-964
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: TRUNK
> Reporter: stack
> Assignee: stack
> Fix For: 0.19.0
>
> Attachments: 964-logs-1.zip
>
>
> Seems easy to reproduce.
> I am trying to start a master then start a regionserver. I'm using local filesystem.
> I start master like this:
> {code}
> ./bin/hbase master start
> {code}
> The regionserver similarly.
> All comes up. Then I try to run sequentialWriter from PE.
> Here is what shows in master log:
> {code}
> ...
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:06 INFO master.BaseScanner: RegionManager.rootScanner scanning meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020}
> 08/10/27 15:20:07 DEBUG master.BaseScanner: RegionManager.rootScanner REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_ROOT => 'false', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}, {NAME => 'historian', BLOOMFILTER => 'false', VERSIONS => '2147483647', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '-1', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}}, SERVER => '208.84.6.64:60020', STARTCODE => 1225145942330
> 08/10/27 15:20:07 INFO master.BaseScanner: RegionManager.rootScanner scan of meta region {regionname: -ROOT-,,0, startKey: <>, server: 208.84.6.64:60020} complete
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> 08/10/27 15:20:08 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:10 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> ...
> {code}
> Here is the regionserver log:
> {code}
> ....
> 08/10/27 15:19:09 DEBUG regionserver.CompactSplitThread: Compaction requested for region: .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: starting compaction on region .META.,,1
> 08/10/27 15:19:09 INFO regionserver.HRegion: compaction completed on region .META.,,1 in 0sec
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 INFO regionserver.HRegionServer: MSG_REGION_OPEN: TestTable,,1225145988632
> 08/10/27 15:19:48 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:50 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:52 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:54 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:56 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:19:58 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:00 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:02 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:04 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:07 DEBUG client.HConnectionManager$TableServers: Sleeping 2000ms, waiting for root region.
> 08/10/27 15:20:09 DEBUG client.HConnectionManager$TableServers: Attempt 0 of 10 failed with <org.apache.hadoop.hbase.client.NoServerForRegionException: Timed out trying to locate root region>. Retrying after sleep of 2000
> ...
> {code}
> and so on.
> Giving to Jim since he was hereabouts most recently.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.