Inconsistent table and how to handle

[茅旭峰 <m9...@gmail.com>]

I start a new thread to make things clearer.

It seems like we have run into a inconsistent table issue.

We are using hadoop-0.20.2-CDH3B4, zookeeper-3.3.2-CDH3B4
and hbase-0.90.1-CDH3B4.

The original issue we've run into is like

====
hbase(main):006:0> put 'table1', 'abc', 'cfEStore:dasd', '123'

0 row(s) in 0.3030 seconds

hbase(main):007:0> put 'table1', 'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=',
'cfEStore:dasd', '123'

ERROR: java.io.IOException: HRegionInfo was null or empty in .META.,
row=keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd./info:serverstartcode/1300672190144/Put/vlen=8}

Here is some help for this command:
Put a cell 'value' at specified table/row/column and optionally
timestamp coordinates.  To put a cell value into table 't1' at
row 'r1' under column 'c1' marked with the time 'ts1', do:

  hbase> put 't1', 'r1', 'c1', 'value', ts1

====

In the result of scan '.META.', we have two regions hold the put key
'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok='
====
#1, 'LC3MILeAUy8HmRFgU5-ESE-9T7w=' -> 'LD4jOJWFyt4m7A3KGFST6d-uj3A='
#2, 'LC_vN8JYweYYsnKaKbpOo67kUNA=' -> 'some end key' actually no end_key
info
====

'LCgwzrx2XTFkB2Ymz9HeJWPY0Ok=' should exists in the first region, but the
second region has the closer start key row.

In ./bin/hbase hbck results, we see

====
  table1,LC3MILeAUy8HmRFgU5-ESE-9T7w=,1300519432575.0bdd3d8fa7fc710860a4ee51fc9c8625.
id: 1300519432575 encoded_name: 0bdd3d8fa7fc710860a4ee51fc9c8625 start:
LC3MILeAUy8HmRFgU5-ESE-9T7w= end: LD4jOJWFyt4m7A3KGFST6d-uj3A=
  table1,LD4jOJWFyt4m7A3KGFST6d-uj3A=,1300583204256.b7d6e116d91d516a0c1709fb29e765b4.
id: 1300583204256 encoded_name: b7d6e116d91d516a0c1709fb29e765b4 start:
LD4jOJWFyt4m7A3KGFST6d-uj3A= end: LDaGUgQLscCTC4771l9Rq4MHORs=
...
Number of empty REGIONINFO_QUALIFIER rows in .META.: 18
  keyvalues={table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.
ba8e0bffb79bda039f6800f78ad22dcd./info:server/1300672190144/Put/vlen=14,
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.
ba8e0bffb79bda039f6800f78ad22dcd
./info:serverstartcode/1300672190144/Put/vlen=8}
...
ERROR: Region hdfs://cloud137:9000/hbase/table1/ba8e0bffb79bda039f6800f78ad22dcd
on HDFS, but not listed in META or deployed on any region server.
====

Some regionserver log shows
====
2011-03-21 09:48:36,953 DEBUG
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing
close of table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.
ba8e0bffb79bda039f6800f78ad22dcd.
2011-03-21 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Closing
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.:
disabling compactions & flushes
2011-03-21 09:48:36,953 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Updates disabled for region
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
2011-03-21 09:48:36,953 INFO org.apache.hadoop.hbase.regionserver.HRegion:
Closed
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
2011-03-21 09:48:36,953 DEBUG
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Closed
region
table1,LC_vN8JYweYYsnKaKbpOo67kUNA=,1300594695996.ba8e0bffb79bda039f6800f78ad22dcd.
====

So maybe the region 'LC_vN8JYweYYsnKaKbpOo67kUNA=' is closed, but the
directory
/hbase/table1/ba8e0bffb79bda039f6800f78ad22dcd  is still there for some
reason.

What confuses me is why the 'hbase hbck' can figure out the region '
LC_vN8JYweYYsnKaKbpOo67kUNA=' is a bad one,
but the master or the result of scan'.META.' can not tell. I think even the
table has run into an inconsistent status,
the 'hbase hbck' and the 'hbase shell' should also has some kind of
consistency.

Now what shall I do to recover this issue?

Mao Xu-Feng