You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Chris Howe <ho...@infochimps.org> on 2011/01/28 06:12:45 UTC
Inconsistent META data for a region.
Howdy,
I was trying to use "add_table.rb" to restore a table that I had copied the hdfs
files for, and I had some trouble. Now when I run "hbase hbck" I get the
following:
...
ERROR: Region test,,1296067171940.0200bfe58a9e9fadf8ebfa523c47332f. found on
server 10.101.45.82:60020 but is listed in META to be on server ip-10-117-86-
81.ec2.internal:60020.
ERROR: Region test,,1296067171940.0200bfe58a9e9fadf8ebfa523c47332f. found on
server 10.100.225.127:60020 but is listed in META to be on server ip-10-117-86-
81.ec2.internal:60020.
Detected 2 inconsistencies. This might not indicate a real problem because these
regions could be in the midst of a split. Consider re-running with a larger
value of -timelag.
...
(I am running Version 0.89.20100924+28, r )
Now, this table is not at all important to me. However, if I drop it, these two
regions end up surviving, and hbck gives me a different inconsistency message
(... not listed in META...).
Is there a way that I can just zot these regions out and make hbase hbck happy
again? I do have a fair amount of data in some other tables that I would rather
not lose.
Also, does any one have any pointers for getting add_table.rb to work properly?
Whenever I run it, it creates the table schema ok, but it seems to miss the mark
on getting the regions registered. They either get assigned to the wrong
regionservers or they dont get assigned to anything. In either case, when I do a
scan of the table, I come up with nothing.
Thanks!
--Chris
Re: Inconsistent META data for a region.
Posted by Chris Howe <ho...@infochimps.org>.
Chris Howe <ho...@...> writes:
>
> I was trying to use "add_table.rb" to restore a table that I had copied the
hdfs
> files for, and I had some trouble. Now when I run "hbase hbck" I get the
> following:
>
> ...
> ERROR: Region test,,1296067171940.0200bfe58a9e9fadf8ebfa523c47332f. found on
> server 10.101.45.82:60020 but is listed in META to be on server ip-10-117-86-
> 81.ec2.internal:60020.
> ERROR: Region test,,1296067171940.0200bfe58a9e9fadf8ebfa523c47332f. found on
> server 10.100.225.127:60020 but is listed in META to be on server ip-10-117-
86-
> 81.ec2.internal:60020.
> Detected 2 inconsistencies. This might not indicate a real problem because
these
> regions could be in the midst of a split. Consider re-running with a larger
> value of -timelag.
> ...
>
I ended up restarting all of the regionservers, and the ghost regions decided to
go away. I am still looking for a way to run add_table.rb that actually works.
--Chris
Re: Inconsistent META data for a region.
Posted by Stack <st...@duboce.net>.
On Fri, Jan 28, 2011 at 10:56 AM, Chris Howe <ho...@infochimps.org> wrote:
> The region had been deployed, but I dropped the table before I tried to re-add
> it.
>
OK. This could have been the cause. Our disable/drop was flakey
pre-0.90. Maybe it failed close out all regions.
> When I would stop a single regionserver that was hosting the phantom region, it
> would just migrate to a different one. I managed to get the warnings to
> completely go away for a time this morning, by shutting down all of the
> regionservers. Hbck came back clean.
>
It'd migrate without updating .META. w/ new location?
> Unfortunately, I then tried loading a large table that I had created as a set of
> HFiles, and when I ran "loadtable" on them, I was rewarded with 281 regions that
> are now on regionservers and not in META.
>
You are on a 0.89.x, are you? If so, should you be using
completebulkload -- see http://hbase.apache.org/bulk-loads.html
(Sorry, there is a bit of overlap between completebulkload and
loadtable that is to be fixed. completebulkload is the future).
>> > Is there a way that I can just zot these regions out and make hbase hbck
> happy
>> > again?
>>
hbck can take a while to settle. It continues to report inconsistency?
Try stopping and starting your cluster. That should take care of it.
Can you go to 0.90.0?
St.Ack
Re: Inconsistent META data for a region.
Posted by Chris Howe <ho...@infochimps.org>.
Stack <st...@...> writes:
> > ERROR: Region test,,1296067171940.0200bfe58a9e9fadf8ebfa523c47332f. found on
> > server 10.101.45.82:60020 but is listed in META to be on server ip-10-117-
86-
> > 81.ec2.internal:60020.
>
> Could this region have been deployed on this server before you ran
> add_table? Or 'test' was a completely new addition.
The region had been deployed, but I dropped the table before I tried to re-add
it.
> If the latter, then this is a new spin. add_table.rb effectively just
> edits .META. Somehow, the assignment of the just-added regions went
> awry. Can you grep this region name in your master log? You might be
> able to make some sense of what happened; was the region assigned two
> places?
>
> > Now, this table is not at all important to me. However, if I drop it, these
two
> > regions end up surviving, and hbck gives me a different inconsistency
message
> > (... not listed in META...).
> >
>
> Yeah. Disable and drop are going by the content in .META. Thats
> supposed to be the authority. You could restart these individual
> regionservers (or restart cluster)? That'd clean up the mess.
When I would stop a single regionserver that was hosting the phantom region, it
would just migrate to a different one. I managed to get the warnings to
completely go away for a time this morning, by shutting down all of the
regionservers. Hbck came back clean.
Unfortunately, I then tried loading a large table that I had created as a set of
HFiles, and when I ran "loadtable" on them, I was rewarded with 281 regions that
are now on regionservers and not in META.
> > Is there a way that I can just zot these regions out and make hbase hbck
happy
> > again?
>
> Not in 0.89.x. I believe you can send a direct close region to the
> individual servers in 0.90 (I'd have to check) so you don't have to do
> full regionserver restart.
I did find the close_region tool in the shell, but I am not sure that it worked
for me.
>
> How did you make the table? Was it with bulk loader?
>
The "test" table was made with "puts". My new disaster is an HTable.
Re: Inconsistent META data for a region.
Posted by Stack <st...@duboce.net>.
On Thu, Jan 27, 2011 at 9:12 PM, Chris Howe <ho...@infochimps.org> wrote:
> Howdy,
>
Howdy back.
See in below.
> ERROR: Region test,,1296067171940.0200bfe58a9e9fadf8ebfa523c47332f. found on
> server 10.101.45.82:60020 but is listed in META to be on server ip-10-117-86-
> 81.ec2.internal:60020.
Could this region have been deployed on this server before you ran
add_table? Or 'test' was a completely new addition.
If the latter, then this is a new spin. add_table.rb effectively just
edits .META. Somehow, the assignment of the just-added regions went
awry. Can you grep this region name in your master log? You might be
able to make some sense of what happened; was the region assigned two
places?
> Now, this table is not at all important to me. However, if I drop it, these two
> regions end up surviving, and hbck gives me a different inconsistency message
> (... not listed in META...).
>
Yeah. Disable and drop are going by the content in .META. Thats
supposed to be the authority. You could restart these individual
regionservers (or restart cluster)? That'd clean up the mess.
> Is there a way that I can just zot these regions out and make hbase hbck happy
> again? I do have a fair amount of data in some other tables that I would rather
> not lose.
>
Not in 0.89.x. I believe you can send a direct close region to the
individual servers in 0.90 (I'd have to check) so you don't have to do
full regionserver restart.
> Also, does any one have any pointers for getting add_table.rb to work properly?
> Whenever I run it, it creates the table schema ok, but it seems to miss the mark
> on getting the regions registered. They either get assigned to the wrong
> regionservers or they dont get assigned to anything. In either case, when I do a
> scan of the table, I come up with nothing.
>
How did you make the table? Was it with bulk loader?
St.Ack