Is there any way I could force record a HBase table that has missing blocks.

[Stanley Xu <we...@gmail.com>]

Dear all,

We were doing some network migration on our system and made some mistakes in
the operation. So for some of our hbase data, we have some block missed,
like the following by a fsck command output(in the end part of the mail). I
am wondering if we could just ignore the missing blocks(let's say we just
lost part of the data) but still keep the whole table available? Because our
backup didn't cover all the data in the hbase table. If we could ignore the
missing blocks, and do an overwrite with the backup data, the data we lost
is trivial.

If we have to drop the table and recover from the file backup, we might lose
some of the column family that we didn't back up in our backup storage
system. Thanks in advance.

Status: CORRUPT
 Total size: 27674055256 B
 Total dirs: 1777
 Total files: 3094
 Total blocks (validated): 3170 (avg. block size 8729985 B)
  ********************************
  CORRUPT FILES: 22
  MISSING BLOCKS: 22
  MISSING SIZE: 176158662 B
  CORRUPT BLOCKS: 22
  ********************************
 Minimally replicated blocks: 3148 (99.30599 %)
 Over-replicated blocks: 0 (0.0 %)
 Under-replicated blocks: 0 (0.0 %)
 Mis-replicated blocks: 0 (0.0 %)
 Default replication factor: 3
 Average block replication: 2.9681387
 Corrupt blocks: 22
 Missing replicas: 0 (0.0 %)
 Number of data-nodes: 37
 Number of racks: 1


The filesystem under path '/hbase/URLTag' is CORRUPT



Best wishes,
Stanley Xu

Re: Is there any way I could force record a HBase table that has missing blocks.

[Stack <st...@duboce.net>]

On Thu, May 5, 2011 at 7:27 AM, Stanley Xu <we...@gmail.com> wrote:
> We were doing some network migration on our system and made some mistakes in
> the operation. So for some of our hbase data, we have some block missed,
> like the following by a fsck command output(in the end part of the mail). I
> am wondering if we could just ignore the missing blocks(let's say we just
> lost part of the data) but still keep the whole table available? Because our
> backup didn't cover all the data in the hbase table. If we could ignore the
> missing blocks, and do an overwrite with the backup data, the data we lost
> is trivial.
>

The blocks belong to a file.  An option on fsck will output the files
that have the missing blocks.  You could remove those files and HBase
should come up or replace them from your backup if they exists
therein.  If the missing blocks are in .META. files and you do not
have a recovery, then getting hbase back up will be a bit more of an
interesting exercise.   We can help if the latter is the case.

St.Ack