region in a bad state - how to manually fix

[Bill Graham <bi...@gmail.com>]

Hi,

We have an empty table that is somehow in a bad state that I'm unable
to disable or drop. We're running 0.90.0 on CDH3b2. Is there a way
that I can manually remove this table from HBase without making a mess
of things?

The table has 2 CFs and it's empty. When I do a scan I get this:

org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Region is not
online: kpd_user_info2,,1300458908181.412ae74e76cc4000da10233b3e6f163e.
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2319)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1764)
        at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036)

And when I try to disable it, the command hangs and this message
appears in the HBase master every 40 seconds:

2011-03-29 14:27:39,373 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Regions in
transition timed out:
kpd_user_info2,,1300458908181.412ae74e76cc4000da10233b3e6f163e.
state=PENDING_CLOSE, ts=1301434029373
2011-03-29 14:27:39,374 INFO
org.apache.hadoop.hbase.master.AssignmentManager: Region has been
PENDING_CLOSE for too long, running forced unassign again on
region=kpd_user_info2,,1300458908181.412ae74e76cc4000da10233b3e6f163e.
2011-03-29 14:27:39,376 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Starting
unassignment of region
kpd_user_info2,,1300458908181.412ae74e76cc4000da10233b3e6f163e.
(offlining)
2011-03-29 14:27:39,376 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Attempting to
unassign region
kpd_user_info2,,1300458908181.412ae74e76cc4000da10233b3e6f163e. which
is already pending close but forcing an additional close
2011-03-29 14:27:39,382 DEBUG
org.apache.hadoop.hbase.master.AssignmentManager: Server
serverName=hadoop-wkr-r52-n1.somedomain.com,60020,1299170041690,
load=(requests=0, regions=0, usedHeap=0, maxHeap=0) returned
org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException: Received close for
kpd_user_info2,,1300458908181.412ae74e76cc4000da10233b3e6f163e. but we
are not serving it for 412ae74e76cc4000da10233b3e6f163e


What would happen if I manually removed it's regions from .META. and
it's directory from HDFS?

thanks,
Bill

Re: region in a bad state - how to manually fix

[Stack <st...@duboce.net>]

On Tue, Mar 29, 2011 at 2:44 PM, Bill Graham <bi...@gmail.com> wrote:
> What would happen if I manually removed it's regions from .META. and
> it's directory from HDFS?
>

Grep kpd_user_info2,,1300458908181.412ae74e76cc4000da10233b3e6f163e.
in master logs

See if its on any server.  If not, go ahead and delete from .META. and
from hdfs.  That should clean it up.

Otherwise, if its open on one of your regionservers.  Try closing it
before doing the above remove from .META.

Use deleteall in the shell to remove all columns for the row.

St.Ack