You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Andy Sautins <an...@returnpath.net> on 2011/05/15 18:11:54 UTC
inconsistencies reported by hbck...
I finally got around to running 'hbase hbck' on our HBase cluster. We noticed the following error being reported:
Chain of regions in table myTable contains less elements than are listed in META; visited=671, edges=681
ERROR: Found inconsistency in table myTable
Can anyone help shed some light on what that error means and how I might resolve it? It does seem like the table is in somewhat of a bad state ( e.g., there are two regions that have blank start key but have different end keys ).
We are running from the cloudera CDH3U0 distribution. Note also this table was originally created and running for a long time under the early .89 releases. I did run both 'hbase hbck -fix' and 'check_meta.rb -fix' and still have the above inconsistency so it looks like I need to take manual action at this point but I'm a little stumped on how to go about it.
Any insight would be appreciated.
Thanks
Andy
Re: inconsistencies reported by hbck...
Posted by Jean-Daniel Cryans <jd...@apache.org>.
So those are the same two regions (region1 in the first error is
region1 in the second error and so on)? If so, it's probably a parent
and a daughter region and somehow (due to a bug, can't tell for sure
what exactly) they were both re-deployed.
To fix, I think you could merge the two regions manually by first
disabling the table, moving the files from one region to the other,
deleting the folder / entry in .META. for the former region, and
finally re-enable the table.
One thing tho is that you have to keep the chain of keys intact
meaning that the end key of region1/2 merged should be the start key
of whatever comes next (let's call it region3).
If you aren't sure which region to choose and you don't want to share
your region names here, feel free to send me the .META. entries
offline so that I can advise you.
Stack has more experience than me doing that sort of fixups so he
might have another idea, but since he works next to me we could figure
it out together as long as we have the region keys :)
Thx,
J-D
On Mon, May 16, 2011 at 12:38 PM, Andy Sautins
<an...@returnpath.net> wrote:
>
> J-D,
>
> I was able to run the 0.90.3 rc hbck on our hbase cluster and it does give more detail. It gives more details. Two specific errors:
>
> - ERROR: (regions <region1> and <region2>) Two regions have the same startkey:
> - ERROR: (regions <region1> and <region2>) There is an overlap in the region chain.
>
> That is more detail, although I'm not really sure what to do with that information. Is there a way to effectively repair from these two errors?
>
> Thanks
>
> Andy
>
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
> Sent: Monday, May 16, 2011 1:05 PM
> To: user@hbase.apache.org
> Subject: Re: inconsistencies reported by hbck...
>
> Would you be able to patch in
> https://issues.apache.org/jira/browse/HBASE-3695 and see what hbck
> tells you now? Else you could try using the 0.90.3 rc0 which has it
> too: http://people.apache.org/~stack/hbase-0.90.3-candidate-0/
>
> J-D
>
> On Sun, May 15, 2011 at 9:11 AM, Andy Sautins
> <an...@returnpath.net> wrote:
>>
>> I finally got around to running 'hbase hbck' on our HBase cluster. We noticed the following error being reported:
>>
>> Chain of regions in table myTable contains less elements than are listed in META; visited=671, edges=681
>> ERROR: Found inconsistency in table myTable
>>
>> Can anyone help shed some light on what that error means and how I might resolve it? It does seem like the table is in somewhat of a bad state ( e.g., there are two regions that have blank start key but have different end keys ).
>>
>> We are running from the cloudera CDH3U0 distribution. Note also this table was originally created and running for a long time under the early .89 releases. I did run both 'hbase hbck -fix' and 'check_meta.rb -fix' and still have the above inconsistency so it looks like I need to take manual action at this point but I'm a little stumped on how to go about it.
>>
>> Any insight would be appreciated.
>>
>> Thanks
>>
>> Andy
>>
>>
>>
>
RE: inconsistencies reported by hbck...
Posted by Andy Sautins <an...@returnpath.net>.
J-D,
I was able to run the 0.90.3 rc hbck on our hbase cluster and it does give more detail. It gives more details. Two specific errors:
- ERROR: (regions <region1> and <region2>) Two regions have the same startkey:
- ERROR: (regions <region1> and <region2>) There is an overlap in the region chain.
That is more detail, although I'm not really sure what to do with that information. Is there a way to effectively repair from these two errors?
Thanks
Andy
-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-Daniel Cryans
Sent: Monday, May 16, 2011 1:05 PM
To: user@hbase.apache.org
Subject: Re: inconsistencies reported by hbck...
Would you be able to patch in
https://issues.apache.org/jira/browse/HBASE-3695 and see what hbck
tells you now? Else you could try using the 0.90.3 rc0 which has it
too: http://people.apache.org/~stack/hbase-0.90.3-candidate-0/
J-D
On Sun, May 15, 2011 at 9:11 AM, Andy Sautins
<an...@returnpath.net> wrote:
>
> I finally got around to running 'hbase hbck' on our HBase cluster. We noticed the following error being reported:
>
> Chain of regions in table myTable contains less elements than are listed in META; visited=671, edges=681
> ERROR: Found inconsistency in table myTable
>
> Can anyone help shed some light on what that error means and how I might resolve it? It does seem like the table is in somewhat of a bad state ( e.g., there are two regions that have blank start key but have different end keys ).
>
> We are running from the cloudera CDH3U0 distribution. Note also this table was originally created and running for a long time under the early .89 releases. I did run both 'hbase hbck -fix' and 'check_meta.rb -fix' and still have the above inconsistency so it looks like I need to take manual action at this point but I'm a little stumped on how to go about it.
>
> Any insight would be appreciated.
>
> Thanks
>
> Andy
>
>
>
Re: inconsistencies reported by hbck...
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Would you be able to patch in
https://issues.apache.org/jira/browse/HBASE-3695 and see what hbck
tells you now? Else you could try using the 0.90.3 rc0 which has it
too: http://people.apache.org/~stack/hbase-0.90.3-candidate-0/
J-D
On Sun, May 15, 2011 at 9:11 AM, Andy Sautins
<an...@returnpath.net> wrote:
>
> I finally got around to running 'hbase hbck' on our HBase cluster. We noticed the following error being reported:
>
> Chain of regions in table myTable contains less elements than are listed in META; visited=671, edges=681
> ERROR: Found inconsistency in table myTable
>
> Can anyone help shed some light on what that error means and how I might resolve it? It does seem like the table is in somewhat of a bad state ( e.g., there are two regions that have blank start key but have different end keys ).
>
> We are running from the cloudera CDH3U0 distribution. Note also this table was originally created and running for a long time under the early .89 releases. I did run both 'hbase hbck -fix' and 'check_meta.rb -fix' and still have the above inconsistency so it looks like I need to take manual action at this point but I'm a little stumped on how to go about it.
>
> Any insight would be appreciated.
>
> Thanks
>
> Andy
>
>
>