You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Dmitriy Lyubimov <dl...@gmail.com> on 2011/02/23 21:02:22 UTC
table creation is failing now and then (CDH3b3)
Hi all,
from time to time we come to a sitation where .META. table seems to be stuck
in some corrupted state.
In particular, attempts to create more tables cause
ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: No server
address listed in .META. for region
LEAD_DATA,,1298484698945.8226d7a44286402e80c9b991d2c00cc3.
The server is not even under any load worth mentioning at the moment.
Table manipulations are failing with both shell and HBAdmin, so it's not an
isolated shell issue (there seems to be similar issue related to shell only,
872, but it's pretty old one).
I scanned for solutions for this particular error, and it seems to be
revolving around flushing .META. , major compact it and restarting the
master(s).
In our case, only restarting the master helped to remove the condition.
logs seem to be stacktrace-free.
My questions are --
-- is there a way to figure out what causes those inconsistences in .META.?
-- is there a way to fix that other than restarting master? Is restarting
master in this case implies a downtime?
Thanks.
-Dmitriy
Re: table creation is failing now and then (CDH3b3)
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Thank you, sir.
On Thu, Feb 24, 2011 at 4:12 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> See http://hbase.apache.org/upgrading.html#upgrade0.90
>
> Also if you want to stay on CDH, CDH3b4 was just released.
>
> J-D
>
> On Thu, Feb 24, 2011 at 4:08 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> > Thank you, sir.
> >
> > Do you happen to know if hbase swtich from 0.89 CDH3b3 to 0.90.1 is going
> to
> > be file-level compatible? I.e. do i have to export and re-import data in
> > between?
> >
> > Thank you.
> >
> >
> > On Wed, Feb 23, 2011 at 12:05 PM, Ryan Rawson <ry...@gmail.com>
> wrote:
> >
> >> You should consider upgrading to hbase 0.90.1, a lot of these kinds of
> >> issues were fixed.
> >>
> >> -ryan
> >>
> >> On Wed, Feb 23, 2011 at 12:02 PM, Dmitriy Lyubimov <dl...@gmail.com>
> >> wrote:
> >> > Hi all,
> >> >
> >> > from time to time we come to a sitation where .META. table seems to be
> >> stuck
> >> > in some corrupted state.
> >> > In particular, attempts to create more tables cause
> >> >
> >> > ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: No
> >> server
> >> > address listed in .META. for region
> >> > LEAD_DATA,,1298484698945.8226d7a44286402e80c9b991d2c00cc3.
> >> >
> >> > The server is not even under any load worth mentioning at the moment.
> >> >
> >> > Table manipulations are failing with both shell and HBAdmin, so it's
> not
> >> an
> >> > isolated shell issue (there seems to be similar issue related to shell
> >> only,
> >> > 872, but it's pretty old one).
> >> >
> >> > I scanned for solutions for this particular error, and it seems to be
> >> > revolving around flushing .META. , major compact it and restarting the
> >> > master(s).
> >> >
> >> > In our case, only restarting the master helped to remove the
> condition.
> >> >
> >> > logs seem to be stacktrace-free.
> >> >
> >> > My questions are --
> >> >
> >> > -- is there a way to figure out what causes those inconsistences in
> >> .META.?
> >> > -- is there a way to fix that other than restarting master? Is
> restarting
> >> > master in this case implies a downtime?
> >> >
> >> > Thanks.
> >> > -Dmitriy
> >> >
> >>
> >
>
Re: table creation is failing now and then (CDH3b3)
Posted by Jean-Daniel Cryans <jd...@apache.org>.
See http://hbase.apache.org/upgrading.html#upgrade0.90
Also if you want to stay on CDH, CDH3b4 was just released.
J-D
On Thu, Feb 24, 2011 at 4:08 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Thank you, sir.
>
> Do you happen to know if hbase swtich from 0.89 CDH3b3 to 0.90.1 is going to
> be file-level compatible? I.e. do i have to export and re-import data in
> between?
>
> Thank you.
>
>
> On Wed, Feb 23, 2011 at 12:05 PM, Ryan Rawson <ry...@gmail.com> wrote:
>
>> You should consider upgrading to hbase 0.90.1, a lot of these kinds of
>> issues were fixed.
>>
>> -ryan
>>
>> On Wed, Feb 23, 2011 at 12:02 PM, Dmitriy Lyubimov <dl...@gmail.com>
>> wrote:
>> > Hi all,
>> >
>> > from time to time we come to a sitation where .META. table seems to be
>> stuck
>> > in some corrupted state.
>> > In particular, attempts to create more tables cause
>> >
>> > ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: No
>> server
>> > address listed in .META. for region
>> > LEAD_DATA,,1298484698945.8226d7a44286402e80c9b991d2c00cc3.
>> >
>> > The server is not even under any load worth mentioning at the moment.
>> >
>> > Table manipulations are failing with both shell and HBAdmin, so it's not
>> an
>> > isolated shell issue (there seems to be similar issue related to shell
>> only,
>> > 872, but it's pretty old one).
>> >
>> > I scanned for solutions for this particular error, and it seems to be
>> > revolving around flushing .META. , major compact it and restarting the
>> > master(s).
>> >
>> > In our case, only restarting the master helped to remove the condition.
>> >
>> > logs seem to be stacktrace-free.
>> >
>> > My questions are --
>> >
>> > -- is there a way to figure out what causes those inconsistences in
>> .META.?
>> > -- is there a way to fix that other than restarting master? Is restarting
>> > master in this case implies a downtime?
>> >
>> > Thanks.
>> > -Dmitriy
>> >
>>
>
Re: table creation is failing now and then (CDH3b3)
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Thank you, sir.
Do you happen to know if hbase swtich from 0.89 CDH3b3 to 0.90.1 is going to
be file-level compatible? I.e. do i have to export and re-import data in
between?
Thank you.
On Wed, Feb 23, 2011 at 12:05 PM, Ryan Rawson <ry...@gmail.com> wrote:
> You should consider upgrading to hbase 0.90.1, a lot of these kinds of
> issues were fixed.
>
> -ryan
>
> On Wed, Feb 23, 2011 at 12:02 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
> > Hi all,
> >
> > from time to time we come to a sitation where .META. table seems to be
> stuck
> > in some corrupted state.
> > In particular, attempts to create more tables cause
> >
> > ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: No
> server
> > address listed in .META. for region
> > LEAD_DATA,,1298484698945.8226d7a44286402e80c9b991d2c00cc3.
> >
> > The server is not even under any load worth mentioning at the moment.
> >
> > Table manipulations are failing with both shell and HBAdmin, so it's not
> an
> > isolated shell issue (there seems to be similar issue related to shell
> only,
> > 872, but it's pretty old one).
> >
> > I scanned for solutions for this particular error, and it seems to be
> > revolving around flushing .META. , major compact it and restarting the
> > master(s).
> >
> > In our case, only restarting the master helped to remove the condition.
> >
> > logs seem to be stacktrace-free.
> >
> > My questions are --
> >
> > -- is there a way to figure out what causes those inconsistences in
> .META.?
> > -- is there a way to fix that other than restarting master? Is restarting
> > master in this case implies a downtime?
> >
> > Thanks.
> > -Dmitriy
> >
>
Re: table creation is failing now and then (CDH3b3)
Posted by Ryan Rawson <ry...@gmail.com>.
You should consider upgrading to hbase 0.90.1, a lot of these kinds of
issues were fixed.
-ryan
On Wed, Feb 23, 2011 at 12:02 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Hi all,
>
> from time to time we come to a sitation where .META. table seems to be stuck
> in some corrupted state.
> In particular, attempts to create more tables cause
>
> ERROR: org.apache.hadoop.hbase.client.NoServerForRegionException: No server
> address listed in .META. for region
> LEAD_DATA,,1298484698945.8226d7a44286402e80c9b991d2c00cc3.
>
> The server is not even under any load worth mentioning at the moment.
>
> Table manipulations are failing with both shell and HBAdmin, so it's not an
> isolated shell issue (there seems to be similar issue related to shell only,
> 872, but it's pretty old one).
>
> I scanned for solutions for this particular error, and it seems to be
> revolving around flushing .META. , major compact it and restarting the
> master(s).
>
> In our case, only restarting the master helped to remove the condition.
>
> logs seem to be stacktrace-free.
>
> My questions are --
>
> -- is there a way to figure out what causes those inconsistences in .META.?
> -- is there a way to fix that other than restarting master? Is restarting
> master in this case implies a downtime?
>
> Thanks.
> -Dmitriy
>