You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Yu...@DELL.com on 2013/03/15 23:45:02 UTC

hbase server incorrect shutdown case table corruption?

Hi,

We recently encountered a few issues about hbase server incorrect shutdown causing table corruption issues. Basically, if the HBase server was not gracefully shutdown using stop-hbase.sh, instead it was shutdown by linux server reboot, we got '.META' table missing table rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover from this issue. However, I was wondering if this is the expected behavior of HBase that when hbase server is not gracefully shutdown, the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?

Thanks,

YuLing

Re: hbase server incorrect shutdown case table corruption?

Posted by Ted Yu <yu...@gmail.com>.
Sounds like a plan.

On Fri, Mar 15, 2013 at 6:22 PM, <Yu...@dell.com> wrote:

> So, I guess we should load the latest 0.94.5 and give it a try...if we
> still see the issue, we should report it as an HBase bug?
>
> Thanks,
>
> YuLing
>
> -----Original Message-----
> From: Sergey Shelukhin [mailto:sergey@hortonworks.com]
> Sent: Friday, March 15, 2013 6:07 PM
> To: user@hbase.apache.org
> Subject: Re: hbase server incorrect shutdown case table corruption?
>
> I think it is not acceptable to assume proper shutdown... if this is a
> real issue we should fix it, unless it's already fixed on trunk.
>
> On Fri, Mar 15, 2013 at 5:50 PM, Ted <yu...@gmail.com> wrote:
>
> > Upgrading to 0.94.5 is definitely recommended.
> >
> > As for inconsistencies in .META. Table, can you avoid improper shutdown ?
> >
> > On Mar 15, 2013, at 5:36 PM, <Yu...@DELL.com> wrote:
> >
> > > Hi Ted,
> > >
> > > We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue
> > > will
> > be gone? I mean, HBase won't have this META inconsistence issue after
> > incorrect shutdown?
> > >
> > > We encountered quite a few major issues with 0.94.2, that's why we
> > picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
> > >
> > > Thanks,
> > >
> > > YuLing
> > >
> > > -----Original Message-----
> > > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > > Sent: Friday, March 15, 2013 3:51 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: hbase server incorrect shutdown case table corruption?
> > >
> > > Which HBase version are you using ?
> > >
> > > Did the HBase server experiencing shutdown carry the .META. table ?
> > >
> > > Thanks
> > >
> > > On Fri, Mar 15, 2013 at 3:45 PM, <Yu...@dell.com> wrote:
> > >
> > >> Hi,
> > >>
> > >> We recently encountered a few issues about hbase server incorrect
> > >> shutdown causing table corruption issues. Basically, if the HBase
> > >> server was not gracefully shutdown using stop-hbase.sh, instead it
> > >> was shutdown by linux server reboot, we got '.META' table missing
> > >> table rows. We were able to use 'hsbk -fixMeta -fixAssignment' to
> > >> recover from this issue. However, I was wondering if this is the
> > >> expected behavior of HBase that when hbase server is not gracefully
> > >> shutdown, the database get into inconsistency state?  Is there any
> > >> way from hbase
> > configuration point of view to avoid this behavior?
> > >>
> > >> Thanks,
> > >>
> > >> YuLing
> > >>
> >
>

RE: hbase server incorrect shutdown case table corruption?

Posted by Yu...@DELL.com.
So, I guess we should load the latest 0.94.5 and give it a try...if we still see the issue, we should report it as an HBase bug?

Thanks,

YuLing

-----Original Message-----
From: Sergey Shelukhin [mailto:sergey@hortonworks.com] 
Sent: Friday, March 15, 2013 6:07 PM
To: user@hbase.apache.org
Subject: Re: hbase server incorrect shutdown case table corruption?

I think it is not acceptable to assume proper shutdown... if this is a real issue we should fix it, unless it's already fixed on trunk.

On Fri, Mar 15, 2013 at 5:50 PM, Ted <yu...@gmail.com> wrote:

> Upgrading to 0.94.5 is definitely recommended.
>
> As for inconsistencies in .META. Table, can you avoid improper shutdown ?
>
> On Mar 15, 2013, at 5:36 PM, <Yu...@DELL.com> wrote:
>
> > Hi Ted,
> >
> > We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue 
> > will
> be gone? I mean, HBase won't have this META inconsistence issue after 
> incorrect shutdown?
> >
> > We encountered quite a few major issues with 0.94.2, that's why we
> picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
> >
> > Thanks,
> >
> > YuLing
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > Sent: Friday, March 15, 2013 3:51 PM
> > To: user@hbase.apache.org
> > Subject: Re: hbase server incorrect shutdown case table corruption?
> >
> > Which HBase version are you using ?
> >
> > Did the HBase server experiencing shutdown carry the .META. table ?
> >
> > Thanks
> >
> > On Fri, Mar 15, 2013 at 3:45 PM, <Yu...@dell.com> wrote:
> >
> >> Hi,
> >>
> >> We recently encountered a few issues about hbase server incorrect 
> >> shutdown causing table corruption issues. Basically, if the HBase 
> >> server was not gracefully shutdown using stop-hbase.sh, instead it 
> >> was shutdown by linux server reboot, we got '.META' table missing 
> >> table rows. We were able to use 'hsbk -fixMeta -fixAssignment' to 
> >> recover from this issue. However, I was wondering if this is the 
> >> expected behavior of HBase that when hbase server is not gracefully 
> >> shutdown, the database get into inconsistency state?  Is there any 
> >> way from hbase
> configuration point of view to avoid this behavior?
> >>
> >> Thanks,
> >>
> >> YuLing
> >>
>

Re: hbase server incorrect shutdown case table corruption?

Posted by Sergey Shelukhin <se...@hortonworks.com>.
I think it is not acceptable to assume proper shutdown... if this is a real
issue we should fix it, unless it's already fixed on trunk.

On Fri, Mar 15, 2013 at 5:50 PM, Ted <yu...@gmail.com> wrote:

> Upgrading to 0.94.5 is definitely recommended.
>
> As for inconsistencies in .META. Table, can you avoid improper shutdown ?
>
> On Mar 15, 2013, at 5:36 PM, <Yu...@DELL.com> wrote:
>
> > Hi Ted,
> >
> > We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue will
> be gone? I mean, HBase won't have this META inconsistence issue after
> incorrect shutdown?
> >
> > We encountered quite a few major issues with 0.94.2, that's why we
> picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
> >
> > Thanks,
> >
> > YuLing
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > Sent: Friday, March 15, 2013 3:51 PM
> > To: user@hbase.apache.org
> > Subject: Re: hbase server incorrect shutdown case table corruption?
> >
> > Which HBase version are you using ?
> >
> > Did the HBase server experiencing shutdown carry the .META. table ?
> >
> > Thanks
> >
> > On Fri, Mar 15, 2013 at 3:45 PM, <Yu...@dell.com> wrote:
> >
> >> Hi,
> >>
> >> We recently encountered a few issues about hbase server incorrect
> >> shutdown causing table corruption issues. Basically, if the HBase
> >> server was not gracefully shutdown using stop-hbase.sh, instead it was
> >> shutdown by linux server reboot, we got '.META' table missing table
> >> rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover
> >> from this issue. However, I was wondering if this is the expected
> >> behavior of HBase that when hbase server is not gracefully shutdown,
> >> the database get into inconsistency state?  Is there any way from hbase
> configuration point of view to avoid this behavior?
> >>
> >> Thanks,
> >>
> >> YuLing
> >>
>

RE: hbase server incorrect shutdown case table corruption?

Posted by Yu...@DELL.com.
It's pretty hard for us to cover all the cases of improper shutdown... even if we try our best to cover all the cases, we can't avoid power outage case on customer sites... 

Thanks,

YuLing

-----Original Message-----
From: Ted [mailto:yuzhihong@gmail.com] 
Sent: Friday, March 15, 2013 5:50 PM
To: user@hbase.apache.org
Cc: <us...@hbase.apache.org>
Subject: Re: hbase server incorrect shutdown case table corruption?

Upgrading to 0.94.5 is definitely recommended. 

As for inconsistencies in .META. Table, can you avoid improper shutdown ?

On Mar 15, 2013, at 5:36 PM, <Yu...@DELL.com> wrote:

> Hi Ted,
> 
> We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue will be gone? I mean, HBase won't have this META inconsistence issue after incorrect shutdown?
> 
> We encountered quite a few major issues with 0.94.2, that's why we picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
> 
> Thanks,
> 
> YuLing
> 
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, March 15, 2013 3:51 PM
> To: user@hbase.apache.org
> Subject: Re: hbase server incorrect shutdown case table corruption?
> 
> Which HBase version are you using ?
> 
> Did the HBase server experiencing shutdown carry the .META. table ?
> 
> Thanks
> 
> On Fri, Mar 15, 2013 at 3:45 PM, <Yu...@dell.com> wrote:
> 
>> Hi,
>> 
>> We recently encountered a few issues about hbase server incorrect 
>> shutdown causing table corruption issues. Basically, if the HBase 
>> server was not gracefully shutdown using stop-hbase.sh, instead it 
>> was shutdown by linux server reboot, we got '.META' table missing 
>> table rows. We were able to use 'hsbk -fixMeta -fixAssignment' to 
>> recover from this issue. However, I was wondering if this is the 
>> expected behavior of HBase that when hbase server is not gracefully 
>> shutdown, the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?
>> 
>> Thanks,
>> 
>> YuLing
>> 

Re: hbase server incorrect shutdown case table corruption?

Posted by Ted <yu...@gmail.com>.
Upgrading to 0.94.5 is definitely recommended. 

As for inconsistencies in .META. Table, can you avoid improper shutdown ?

On Mar 15, 2013, at 5:36 PM, <Yu...@DELL.com> wrote:

> Hi Ted,
> 
> We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue will be gone? I mean, HBase won't have this META inconsistence issue after incorrect shutdown?
> 
> We encountered quite a few major issues with 0.94.2, that's why we picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?
> 
> Thanks,
> 
> YuLing
> 
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com] 
> Sent: Friday, March 15, 2013 3:51 PM
> To: user@hbase.apache.org
> Subject: Re: hbase server incorrect shutdown case table corruption?
> 
> Which HBase version are you using ?
> 
> Did the HBase server experiencing shutdown carry the .META. table ?
> 
> Thanks
> 
> On Fri, Mar 15, 2013 at 3:45 PM, <Yu...@dell.com> wrote:
> 
>> Hi,
>> 
>> We recently encountered a few issues about hbase server incorrect 
>> shutdown causing table corruption issues. Basically, if the HBase 
>> server was not gracefully shutdown using stop-hbase.sh, instead it was 
>> shutdown by linux server reboot, we got '.META' table missing table 
>> rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover 
>> from this issue. However, I was wondering if this is the expected 
>> behavior of HBase that when hbase server is not gracefully shutdown, 
>> the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?
>> 
>> Thanks,
>> 
>> YuLing
>> 

RE: hbase server incorrect shutdown case table corruption?

Posted by Yu...@DELL.com.
Hi Ted,

We're using 0.94.1. Do you think if we upgrade to 0.94.5, the issue will be gone? I mean, HBase won't have this META inconsistence issue after incorrect shutdown?

We encountered quite a few major issues with 0.94.2, that's why we picked 0.94.1. Does that mean 0.94.5 is a pretty stable load?

Thanks,

YuLing

-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com] 
Sent: Friday, March 15, 2013 3:51 PM
To: user@hbase.apache.org
Subject: Re: hbase server incorrect shutdown case table corruption?

Which HBase version are you using ?

Did the HBase server experiencing shutdown carry the .META. table ?

Thanks

On Fri, Mar 15, 2013 at 3:45 PM, <Yu...@dell.com> wrote:

> Hi,
>
> We recently encountered a few issues about hbase server incorrect 
> shutdown causing table corruption issues. Basically, if the HBase 
> server was not gracefully shutdown using stop-hbase.sh, instead it was 
> shutdown by linux server reboot, we got '.META' table missing table 
> rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover 
> from this issue. However, I was wondering if this is the expected 
> behavior of HBase that when hbase server is not gracefully shutdown, 
> the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?
>
> Thanks,
>
> YuLing
>

RE: hbase server incorrect shutdown case table corruption?

Posted by Yu...@DELL.com.
I'm using 0.94.1.

We noticed that after HBase server shutdown, the .META. table lost a few rows of the tables. I had to use hbck to repair the .META. to get those rows back.  I was wondering if there could be any configuration that we didn't do properly to cause this issue...

Thanks,

YuLing

-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com] 
Sent: Friday, March 15, 2013 3:51 PM
To: user@hbase.apache.org
Subject: Re: hbase server incorrect shutdown case table corruption?

Which HBase version are you using ?

Did the HBase server experiencing shutdown carry the .META. table ?

Thanks

On Fri, Mar 15, 2013 at 3:45 PM, <Yu...@dell.com> wrote:

> Hi,
>
> We recently encountered a few issues about hbase server incorrect 
> shutdown causing table corruption issues. Basically, if the HBase 
> server was not gracefully shutdown using stop-hbase.sh, instead it was 
> shutdown by linux server reboot, we got '.META' table missing table 
> rows. We were able to use 'hsbk -fixMeta -fixAssignment' to recover 
> from this issue. However, I was wondering if this is the expected 
> behavior of HBase that when hbase server is not gracefully shutdown, 
> the database get into inconsistency state?  Is there any way from hbase configuration point of view to avoid this behavior?
>
> Thanks,
>
> YuLing
>

Re: hbase server incorrect shutdown case table corruption?

Posted by Ted Yu <yu...@gmail.com>.
Which HBase version are you using ?

Did the HBase server experiencing shutdown carry the .META. table ?

Thanks

On Fri, Mar 15, 2013 at 3:45 PM, <Yu...@dell.com> wrote:

> Hi,
>
> We recently encountered a few issues about hbase server incorrect shutdown
> causing table corruption issues. Basically, if the HBase server was not
> gracefully shutdown using stop-hbase.sh, instead it was shutdown by linux
> server reboot, we got '.META' table missing table rows. We were able to use
> 'hsbk -fixMeta -fixAssignment' to recover from this issue. However, I was
> wondering if this is the expected behavior of HBase that when hbase server
> is not gracefully shutdown, the database get into inconsistency state?  Is
> there any way from hbase configuration point of view to avoid this behavior?
>
> Thanks,
>
> YuLing
>