You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Ted Yu <yu...@gmail.com> on 2010/11/23 08:25:38 UTC
disabling table timed out
Hi
We use 0.20.6
I tried to disable packageindex table. From master log:
2010-11-23 07:21:06,326 DEBUG
org.apache.hadoop.hbase.master.ChangeTableState: Adding region
packageindex,CC7E6FEA4CDCF19C6F4AC9BB51EF6A33,1290230596786 to setClosing
list for us01-ciqps1-grid10.carrieriq.com,60020,1290493641949
2010-11-23 07:21:06,326 DEBUG
org.apache.hadoop.hbase.master.ChangeTableState: Adding region
packageindex,F2A18967F48C9FDA9C23BF9A8210ED17,1290230394345 to setClosing
list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
2010-11-23 07:21:06,326 DEBUG
org.apache.hadoop.hbase.master.ChangeTableState: Adding region
packageindex,E8FA713B2F030EF012E5AB0A641CB1DB,1290230356969 to setClosing
list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
2010-11-23 07:21:06,327 DEBUG
org.apache.hadoop.hbase.master.ChangeTableState: Adding region
packageindex,5B10CA26DCAEFBFF4A63DB7D0432D628,1290229869191 to setClosing
list for us01-ciqps1-grid12.carrieriq.com,60020,1290493641232
2010-11-23 07:21:20,178 INFO org.apache.hadoop.hbase.master.ServerManager:
15 region servers, 0 dead, average load 123.66666666666667
2010-11-23 07:21:20,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 10.202.50.111:60020,
regionname: -ROOT-,,0, startKey: <>}
2010-11-23 07:21:20,257 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scan of 1 row(s) of meta region {server:
10.202.50.111:60020, regionname: -ROOT-,,0, startKey: <>} complete
2010-11-23 07:21:22,838 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {server: 10.202.50.101:60020,
regionname: .META.,,1, startKey: <>}
2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scan of 2086 row(s) of meta region {server:
10.202.50.101:60020, regionname: .META.,,1, startKey: <>} complete
2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
But I always got:
hbase(main):004:0> disable 'packageindex'
NativeException: org.apache.hadoop.hbase.RegionException: Retries exhausted,
it took too long to wait for the table packageindex to be disabled.
What should I do to disable the table ?
Thanks
Re: disabling table timed out
Posted by Ted Yu <yu...@gmail.com>.
I tried to restart hbase. But the region server identified by ZNode
/hbase/root-region-server declared that it is not serving root region:
2010-11-23 13:26:49,617 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
locateRegionInMeta attempt 1 of 3 failed; retrying after sleep of 5000
because: Timed out trying to locate root region because:
org.apache.hadoop.hbase.NotServingRegionException: -ROOT-,,0
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2274)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionInfo(HRegionServer.java:1711)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:998)
2010-11-23 13:26:54,622 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
/hbase/root-region-server got 10.202.50.111:60020
2010-11-23 13:26:54,624 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
location changed. Sleeping.
2010-11-23 13:26:59,626 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
finding root region.
2010-11-23 13:26:59,629 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
/hbase/root-region-server got 10.202.50.111:60020
2010-11-23 13:26:59,630 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Root region
location changed. Sleeping.
2010-11-23 13:27:04,632 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Wake. Retry
finding root region.
2010-11-23 13:27:04,635 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
/hbase/root-region-server got 10.202.50.111:60020
What should I do next ?
Thanks
On Tue, Nov 23, 2010 at 1:37 AM, Lars George <la...@gmail.com> wrote:
> Hi Ted,
>
> So one of the regions is not being released? Could you try and see
> from .META. which is still deployed and use the shell's "close_region"
> to close it while looking at the master and region server logs to see
> what is going on? Maybe best if you switch the RS to DEBUG level
> logging first to get some info?
>
> Lars
>
> On Tue, Nov 23, 2010 at 8:25 AM, Ted Yu <yu...@gmail.com> wrote:
> > Hi
> > We use 0.20.6
> >
> > I tried to disable packageindex table. From master log:
> >
> > 2010-11-23 07:21:06,326 DEBUG
> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> > packageindex,CC7E6FEA4CDCF19C6F4AC9BB51EF6A33,1290230596786 to setClosing
> > list for us01-ciqps1-grid10.carrieriq.com,60020,1290493641949
> > 2010-11-23 07:21:06,326 DEBUG
> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> > packageindex,F2A18967F48C9FDA9C23BF9A8210ED17,1290230394345 to setClosing
> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
> > 2010-11-23 07:21:06,326 DEBUG
> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> > packageindex,E8FA713B2F030EF012E5AB0A641CB1DB,1290230356969 to setClosing
> > list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
> > 2010-11-23 07:21:06,327 DEBUG
> > org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> > packageindex,5B10CA26DCAEFBFF4A63DB7D0432D628,1290229869191 to setClosing
> > list for us01-ciqps1-grid12.carrieriq.com,60020,1290493641232
> > 2010-11-23 07:21:20,178 INFO
> org.apache.hadoop.hbase.master.ServerManager:
> > 15 region servers, 0 dead, average load 123.66666666666667
> > 2010-11-23 07:21:20,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
> > RegionManager.rootScanner scanning meta region {server:
> 10.202.50.111:60020,
> > regionname: -ROOT-,,0, startKey: <>}
> > 2010-11-23 07:21:20,257 INFO org.apache.hadoop.hbase.master.BaseScanner:
> > RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> > 10.202.50.111:60020, regionname: -ROOT-,,0, startKey: <>} complete
> > 2010-11-23 07:21:22,838 INFO org.apache.hadoop.hbase.master.BaseScanner:
> > RegionManager.metaScanner scanning meta region {server:
> 10.202.50.101:60020,
> > regionname: .META.,,1, startKey: <>}
> > 2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner:
> > RegionManager.metaScanner scan of 2086 row(s) of meta region {server:
> > 10.202.50.101:60020, regionname: .META.,,1, startKey: <>} complete
> > 2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner:
> All
> > 1 .META. region(s) scanned
> >
> > But I always got:
> > hbase(main):004:0> disable 'packageindex'
> > NativeException: org.apache.hadoop.hbase.RegionException: Retries
> exhausted,
> > it took too long to wait for the table packageindex to be disabled.
> >
> > What should I do to disable the table ?
> >
> > Thanks
> >
>
Re: disabling table timed out
Posted by Lars George <la...@gmail.com>.
Hi Ted,
So one of the regions is not being released? Could you try and see
from .META. which is still deployed and use the shell's "close_region"
to close it while looking at the master and region server logs to see
what is going on? Maybe best if you switch the RS to DEBUG level
logging first to get some info?
Lars
On Tue, Nov 23, 2010 at 8:25 AM, Ted Yu <yu...@gmail.com> wrote:
> Hi
> We use 0.20.6
>
> I tried to disable packageindex table. From master log:
>
> 2010-11-23 07:21:06,326 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> packageindex,CC7E6FEA4CDCF19C6F4AC9BB51EF6A33,1290230596786 to setClosing
> list for us01-ciqps1-grid10.carrieriq.com,60020,1290493641949
> 2010-11-23 07:21:06,326 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> packageindex,F2A18967F48C9FDA9C23BF9A8210ED17,1290230394345 to setClosing
> list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
> 2010-11-23 07:21:06,326 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> packageindex,E8FA713B2F030EF012E5AB0A641CB1DB,1290230356969 to setClosing
> list for us01-ciqps1-grid11.carrieriq.com,60020,1290493641228
> 2010-11-23 07:21:06,327 DEBUG
> org.apache.hadoop.hbase.master.ChangeTableState: Adding region
> packageindex,5B10CA26DCAEFBFF4A63DB7D0432D628,1290229869191 to setClosing
> list for us01-ciqps1-grid12.carrieriq.com,60020,1290493641232
> 2010-11-23 07:21:20,178 INFO org.apache.hadoop.hbase.master.ServerManager:
> 15 region servers, 0 dead, average load 123.66666666666667
> 2010-11-23 07:21:20,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scanning meta region {server: 10.202.50.111:60020,
> regionname: -ROOT-,,0, startKey: <>}
> 2010-11-23 07:21:20,257 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.rootScanner scan of 1 row(s) of meta region {server:
> 10.202.50.111:60020, regionname: -ROOT-,,0, startKey: <>} complete
> 2010-11-23 07:21:22,838 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scanning meta region {server: 10.202.50.101:60020,
> regionname: .META.,,1, startKey: <>}
> 2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner:
> RegionManager.metaScanner scan of 2086 row(s) of meta region {server:
> 10.202.50.101:60020, regionname: .META.,,1, startKey: <>} complete
> 2010-11-23 07:21:24,731 INFO org.apache.hadoop.hbase.master.BaseScanner: All
> 1 .META. region(s) scanned
>
> But I always got:
> hbase(main):004:0> disable 'packageindex'
> NativeException: org.apache.hadoop.hbase.RegionException: Retries exhausted,
> it took too long to wait for the table packageindex to be disabled.
>
> What should I do to disable the table ?
>
> Thanks
>