You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Cosmin Lehene <cl...@adobe.com> on 2009/04/23 11:53:10 UTC
Recovering .META. table
Hi,
Is there any way to rebuild the .META. Table with information in HDFS?
Long story short, the namenode ran out of disk and crashed and we couldn't
have it load the edits.new file at startup (the edits.new file seemed
corrupted and it failed to read some Long value) so we removed the edits.new
file. Apparently the edits file had 10 - 15 entries. Some edit data on
-ROOT-, some edit data on .META., some data on a table compaction
(comapction.dir) and some data on some hbase logs. Only last entry seem to
be bogus, having almost a megabyte of trailing zeroes.
After renaming edits.new HDFS started ok and HBase data is present. However
when we started Hbase one table was missing entirely and one table was
empty. The HBase master report shows Regions On FS: 94, but the report on
region servers shows only 10 regions on all the region servers.
Is there any way to cleanup hbase and rebuild the .META. table from HDFS?
This would be ok even if it implies throwing up some data.
If we would clean the edits.new file and restart HDFS or if we would
manually apply the edits would that still make consistent changes to HBase
data?
Thanks,
Cosmin
Re: Recovering .META. table
Posted by stack <st...@duboce.net>.
On Mon, Apr 27, 2009 at 11:46 AM, Cosmin Lehene <cl...@adobe.com> wrote:
>
>
>
> Having enough info to regenerate the .META. table would be indeed useful.
I don't think it would be too hard to do. We'll try and have something in
place for 0.20.0.
> I wonder if it wouldn't make sense to keep regular snapshots of .META.
> Table as a backup - but I guess this something that the "fsck" tool would
> take care of, right?
The shell allows you force a flush now on a table or individual region.
Could have a cron script invoke the flush then do a copy of the .META.
content that is in hdfs. Better would be a working sync/flush in hdfs
(smile).
I'm glad you got stuff running again Cosmin.
St.Ack
Re: Recovering .META. table
Posted by Cosmin Lehene <cl...@adobe.com>.
Thanks.
We are using Hbase 0.19.1 without DEBUG level :)
I guess it would be possible to read (low level) each region in hdfs and write it to a new table if really necessary, but fortunately we can repopulate the data from scratch.
Having enough info to regenerate the .META. table would be indeed useful. I wonder if it wouldn't make sense to keep regular snapshots of .META. Table as a backup - but I guess this something that the "fsck" tool would take care of, right?
Thanks again,
Cosmin
On 4/24/09 1:49 AM, "stack" <st...@duboce.net> wrote:
On Thu, Apr 23, 2009 at 2:53 AM, Cosmin Lehene <cl...@adobe.com> wrote:
> Hi,
>
> Is there any way to rebuild the .META. Table with information in HDFS?
>
There is not enough info in HDFS to fixup .META. Directory names are
encodings of regionname. There is not a means of going from the encoding to
original. Even then, you'd need to get the HRegionInfo from somewhere.
Currently its only in .META. HRegionInfo has vitals like start and end key,
whether region was offlined.
That rebuild of .META. is not possible using the persisted info is a glaring
omission on our part. We have an issue for an hbase fsck tool but its not
done yet.
If only you had a listing of the .META. content from a time close to the
fillup. In not too recent past I had such a thing and parsed it repairing
a mangled table rename where a script of mine went awry.
What version of hbase Cosmin? In 0.18.x, if at DEBUG level, we'd dump
content of .META. tables to the log. Was turned off in 0.19.x though. Now
we just log changes. Maybe it be possible to parse the master log to
extract all changes and use this as a source to help repair .META.? (I just
took a look, I don't think there is enough info there -- I'll make an
issue).
St.Ack
>
> Long story short, the namenode ran out of disk and crashed and we couldn't
> have it load the edits.new file at startup (the edits.new file seemed
> corrupted and it failed to read some Long value) so we removed the
> edits.new
> file. Apparently the edits file had 10 - 15 entries. Some edit data on
> -ROOT-, some edit data on .META., some data on a table compaction
> (comapction.dir) and some data on some hbase logs. Only last entry seem to
> be bogus, having almost a megabyte of trailing zeroes.
>
> After renaming edits.new HDFS started ok and HBase data is present. However
> when we started Hbase one table was missing entirely and one table was
> empty. The HBase master report shows Regions On FS: 94, but the report on
> region servers shows only 10 regions on all the region servers.
>
> Is there any way to cleanup hbase and rebuild the .META. table from HDFS?
> This would be ok even if it implies throwing up some data.
>
> If we would clean the edits.new file and restart HDFS or if we would
> manually apply the edits would that still make consistent changes to HBase
> data?
>
> Thanks,
> Cosmin
>
>
Re: Recovering .META. table
Posted by stack <st...@duboce.net>.
On Thu, Apr 23, 2009 at 2:53 AM, Cosmin Lehene <cl...@adobe.com> wrote:
> Hi,
>
> Is there any way to rebuild the .META. Table with information in HDFS?
>
There is not enough info in HDFS to fixup .META. Directory names are
encodings of regionname. There is not a means of going from the encoding to
original. Even then, you'd need to get the HRegionInfo from somewhere.
Currently its only in .META. HRegionInfo has vitals like start and end key,
whether region was offlined.
That rebuild of .META. is not possible using the persisted info is a glaring
omission on our part. We have an issue for an hbase fsck tool but its not
done yet.
If only you had a listing of the .META. content from a time close to the
fillup. In not too recent past I had such a thing and parsed it repairing
a mangled table rename where a script of mine went awry.
What version of hbase Cosmin? In 0.18.x, if at DEBUG level, we'd dump
content of .META. tables to the log. Was turned off in 0.19.x though. Now
we just log changes. Maybe it be possible to parse the master log to
extract all changes and use this as a source to help repair .META.? (I just
took a look, I don't think there is enough info there -- I'll make an
issue).
St.Ack
>
> Long story short, the namenode ran out of disk and crashed and we couldn't
> have it load the edits.new file at startup (the edits.new file seemed
> corrupted and it failed to read some Long value) so we removed the
> edits.new
> file. Apparently the edits file had 10 - 15 entries. Some edit data on
> -ROOT-, some edit data on .META., some data on a table compaction
> (comapction.dir) and some data on some hbase logs. Only last entry seem to
> be bogus, having almost a megabyte of trailing zeroes.
>
> After renaming edits.new HDFS started ok and HBase data is present. However
> when we started Hbase one table was missing entirely and one table was
> empty. The HBase master report shows Regions On FS: 94, but the report on
> region servers shows only 10 regions on all the region servers.
>
> Is there any way to cleanup hbase and rebuild the .META. table from HDFS?
> This would be ok even if it implies throwing up some data.
>
> If we would clean the edits.new file and restart HDFS or if we would
> manually apply the edits would that still make consistent changes to HBase
> data?
>
> Thanks,
> Cosmin
>
>