You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zookeeper.apache.org by Ted Yu <yu...@gmail.com> on 2010/11/23 19:19:00 UTC
Fwd: problem starting HBase
Just in case you know how I can get past the following problem.
I specified hbase.zookeeper.property.dataDir with a new directory and the
issue persisted.
Thanks
---------- Forwarded message ----------
From: Ted Yu <yu...@gmail.com>
Date: Tue, Nov 23, 2010 at 7:40 AM
Subject: problem starting HBase
To: dev@hbase.apache.org
I backed up zookeeper dataDir to another location.
After clearing zookeeper dataDir, HMaster still couldn't start:
2010-11-23 15:30:56,095 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master
got 10.202.50.100:60000
2010-11-23 15:30:56,119 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /hbase/root-region-server
2010-11-23 15:30:56,120 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping
5000ms, waiting for root region.
2010-11-23 15:31:01,125 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for /hbase/root-region-server
2010-11-23 15:31:01,125 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Sleeping
5000ms, waiting for root region.
Disk isn't full:
/dev/md2 2786058952 186234928 2456017408 8% /
Comment is appreciated.
On Tue, Nov 23, 2010 at 5:34 AM, Ted Yu <yu...@gmail.com> wrote:
> I tried to restart hbase. But the region server identified by ZNode
> /hbase/root-region-server declared that it is not serving root region:
>
> 2010-11-23 13:26:49,617 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> locateRegionInMeta attempt 1 of 3 failed; retrying after sleep of 5000
> because: Timed out trying to locate root region because:
> org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2274)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1711)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:998)
>
> 2010-11-23 13:26:54,622 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.202.50.111:60020
> 2010-11-23 13:26:54,624 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> 2010-11-23 13:26:59,626 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> 2010-11-23 13:26:59,629 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.202.50.111:60020
> 2010-11-23 13:26:59,630 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
> location changed. Sleeping.
> 2010-11-23 13:27:04,632 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
> finding root region.
> 2010-11-23 13:27:04,635 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
> /hbase/root-region-server got 10.202.50.111:60020
>
> What should I do next ?
>
> Thanks
>
>
> On Tue, Nov 23, 2010 at 1:37 AM, Lars George <la...@gmail.com>wrote:
>
>> Hi Ted,
>>
>> So one of the regions is not being released? Could you try and see
>> from .META. which is still deployed and use the shell's "close_region"
>> to close it while looking at the master and region server logs to see
>> what is going on? Maybe best if you switch the RS to DEBUG level
>> logging first to get some info?
>>
>> Lars
>>
>> On Tue, Nov 23, 2010 at 8:25 AM, Ted Yu <yu...@gmail.com> wrote:
>> > Hi
>> > We use 0.20.6
>> >
>> > I tried to disable packageindex table. From master log:
>> >
>> > 2010-11-23 07:21:06,326 DEBUG
>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
>> > packageindex,CC7E6FEA4CDCF19C6F4AC9BB51EF6A33,1290230596786 to
>> setClosing
>> > list for us01-ciqps1-grid10.carrieriq.com,60020,1290493641949
>> > 2010-11-23 07:21:06,326 DEBUG
>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
>> > packageindex,F2A18967F48C9FDA9C23BF9A8210ED17,1290230394345 to
>> setClosing
>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
>> > 2010-11-23 07:21:06,326 DEBUG
>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
>> > packageindex,E8FA713B2F030EF012E5AB0A641CB1DB,1290230356969 to
>> setClosing
>> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
>> > 2010-11-23 07:21:06,327 DEBUG
>> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
>> > packageindex,5B10CA26DCAEFBFF4A63DB7D0432D628,1290229869191 to
>> setClosing
>> > list for us01-ciqps1-grid12.carrieriq.com,60020,1290493641232
>> > 2010-11-23 07:21:20,178 INFO
>> org.apache.hadoop.hbase.master.ServerManager:
>> > 15 region servers, 0 dead, average load 123.66666666666667
>> > 2010-11-23 07:21:20,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> > RegionManager.rootScanner scanning meta region {server:
>> 10.202.50.111:60020,
>> > regionname: -ROOT-,,0, startKey: <>}
>> > 2010-11-23 07:21:20,257 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> > RegionManager.rootScanner scan of 1 row(s) of meta region {server:
>> > 10.202.50.111:60020, regionname: -ROOT-,,0, startKey: <>} complete
>> > 2010-11-23 07:21:22,838 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> > RegionManager.metaScanner scanning meta region {server:
>> 10.202.50.101:60020,
>> > regionname: .META.,,1, startKey: <>}
>> > 2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> > RegionManager.metaScanner scan of 2086 row(s) of meta region {server:
>> > 10.202.50.101:60020, regionname: .META.,,1, startKey: <>} complete
>> > 2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner:
>> All
>> > 1 .META. region(s) scanned
>> >
>> > But I always got:
>> > hbase(main):004:0> disable 'packageindex'
>> > NativeException: org.apache.hadoop.hbase.RegionException: Retries
>> exhausted,
>> > it took too long to wait for the table packageindex to be disabled.
>> >
>> > What should I do to disable the table ?
>> >
>> > Thanks
>> >
>>
>
>