You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Jean-Daniel Cryans <jd...@apache.org> on 2010/08/31 01:39:06 UTC
[VOTE] Release 'development release' HBase 0.89.2010830 rc1?
It's time for another DR since the new master code is about to be
merged in and we have a few fixes and improvements that'd beneficiate
from more exposure. I branched from trunk this morning and created a
new tag. (See http://wiki.apache.org/hadoop/Hbase/HBaseVersions for
more on what these 0.89.x releases are all about)
Source binary and source tar balls are available here:
http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/
You can also browse the candidate documentation here:
http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/hbase-0.89.20100830/docs/
Issues resolved since 0.89.20100726, our second 0.89.x release, are
roughly ~23 issues odd including fixed deadlocks, better handling of
IOEs during splits and improvements for filters: see
http://su.pr/2HwiUe
Shall we release this candidate as the third in our 0.89.x series of
developer releases?
Please see previous threads on 0.89 releases for more information
about the purpose of this release candidate - in particular, this
'developer release' is for those who can tolerate risk and who are
willing to give feedback in advance of our next major release. We're
not making any guarantees that this is bug free. Its definitely not
for production deploys.
We'll do another release like this in a few weeks after the new master
code has gone in.
Please vote by Monday, September 6th.
Thanks,
J-D
Re: [VOTE] Release 'development release' HBase 0.89.2010830 rc1?
Posted by Stack <st...@duboce.net>.
On Mon, Sep 6, 2010 at 9:32 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On Mon, Sep 6, 2010 at 9:19 PM, Stack <st...@duboce.net> wrote:
> Looking at the diff of HBASE-2461, I think the difference is that we used to
> close the region and then do the writes into META for the new info. Since
> the region was already marked closed by the time we were writing, we
> wouldn't have handlers stuck on the lock. Post-HBASE-2461, the META writes
> happen under the lock before the region gets marked closed.
>
This would explain it.
New axiom: "Never hold a lock while trying to talk to another server"
Lets do new RC (as suggested up on IRC).
St.Ack
Re: [VOTE] Release 'development release' HBase 0.89.2010830 rc1?
Posted by Todd Lipcon <to...@cloudera.com>.
On Mon, Sep 6, 2010 at 9:19 PM, Stack <st...@duboce.net> wrote:
> Nice one Todd. Its a little like HBASE-2880. The issue you found
> looks like it has a few guises. I also think it an issue we've always
> had, not a regression.
>
I don't think so. I killed the server that hit the issue, and within a
couple minutes it happened on another server. I've never seen this before
and this is the same sort of test that I've run on every other release.
Looking at the diff of HBASE-2461, I think the difference is that we used to
close the region and then do the writes into META for the new info. Since
the region was already marked closed by the time we were writing, we
wouldn't have handlers stuck on the lock. Post-HBASE-2461, the META writes
happen under the lock before the region gets marked closed.
>
> Things are different since new master commit last week -- there are
> multiple executors per operation type, open, close, etc., with root
> and meta having their own instantiations -- but you need a handler
> before you get an executor. The case where all handlers could end up
> occupied, in this case 'blocked' on a particular region lock, and no
> progress can be made because an edit needs to be made back into the
> same server before we can progress looks like it can still happen,
> even in the new regime.
>
> I agree we should just keep moving forward. Lets fix in the next 0.89.x.
>
Are other people not seeing this issue under a load test? I got it within an
hour using YCSB with 1G split sizes - I imagine if you tune it down to a
stress-test config with lots of splits, you'll see it even faster.
-Todd
> On Mon, Sep 6, 2010 at 6:20 PM, Todd Lipcon <to...@cloudera.com> wrote:
> > I did some load tests on this afternoon and ran into this bug:
> >
> > https://issues.apache.org/jira/browse/HBASE-2964
> >
> > I got this after loading only 44GB (with 1G region size), so I don't
> think
> > it's that rare. I'm also running with something like 30 handlers. My
> tests
> > are with 80 concurrent clients.
> >
> > So consider me a -0 - if no one else runs into this bug we may as well
> > release, but it seems a step backward in stability for me from the
> 20100621
> > (the last one I did significant load testing on)
> >
> > -Todd
> >
> > On Fri, Sep 3, 2010 at 12:05 PM, Stack <st...@duboce.net> wrote:
> >
> >> +1
> >>
> >> Its running in production at SU.
> >>
> >> St.Ack
> >>
> >> On Mon, Aug 30, 2010 at 5:29 PM, Jean-Daniel Cryans <
> jdcryans@apache.org>
> >> wrote:
> >> > BTW I'm +1, I ran the unit tests, did cluster testing with YCSB, and
> >> > deployed it in almost all our environments here (this will be
> >> > completed before the voting period ends).
> >> >
> >> > J-D
> >> >
> >> > On Mon, Aug 30, 2010 at 4:39 PM, Jean-Daniel Cryans <
> jdcryans@apache.org>
> >> wrote:
> >> >> It's time for another DR since the new master code is about to be
> >> >> merged in and we have a few fixes and improvements that'd beneficiate
> >> >> from more exposure. I branched from trunk this morning and created a
> >> >> new tag. (See http://wiki.apache.org/hadoop/Hbase/HBaseVersions for
> >> >> more on what these 0.89.x releases are all about)
> >> >>
> >> >> Source binary and source tar balls are available here:
> >> >>
> >> >> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/
> >> >>
> >> >> You can also browse the candidate documentation here:
> >> >>
> >> >>
> >>
> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/hbase-0.89.20100830/docs/
> >> >>
> >> >> Issues resolved since 0.89.20100726, our second 0.89.x release, are
> >> >> roughly ~23 issues odd including fixed deadlocks, better handling of
> >> >> IOEs during splits and improvements for filters: see
> >> >> http://su.pr/2HwiUe
> >> >>
> >> >> Shall we release this candidate as the third in our 0.89.x series of
> >> >> developer releases?
> >> >>
> >> >> Please see previous threads on 0.89 releases for more information
> >> >> about the purpose of this release candidate - in particular, this
> >> >> 'developer release' is for those who can tolerate risk and who are
> >> >> willing to give feedback in advance of our next major release. We're
> >> >> not making any guarantees that this is bug free. Its definitely not
> >> >> for production deploys.
> >> >>
> >> >> We'll do another release like this in a few weeks after the new
> master
> >> >> code has gone in.
> >> >>
> >> >> Please vote by Monday, September 6th.
> >> >>
> >> >> Thanks,
> >> >> J-D
> >> >>
> >> >
> >>
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: [VOTE] Release 'development release' HBase 0.89.2010830 rc1?
Posted by Stack <st...@duboce.net>.
Nice one Todd. Its a little like HBASE-2880. The issue you found
looks like it has a few guises. I also think it an issue we've always
had, not a regression.
Things are different since new master commit last week -- there are
multiple executors per operation type, open, close, etc., with root
and meta having their own instantiations -- but you need a handler
before you get an executor. The case where all handlers could end up
occupied, in this case 'blocked' on a particular region lock, and no
progress can be made because an edit needs to be made back into the
same server before we can progress looks like it can still happen,
even in the new regime.
I agree we should just keep moving forward. Lets fix in the next 0.89.x.
Good stuff,
St.Ack
On Mon, Sep 6, 2010 at 6:20 PM, Todd Lipcon <to...@cloudera.com> wrote:
> I did some load tests on this afternoon and ran into this bug:
>
> https://issues.apache.org/jira/browse/HBASE-2964
>
> I got this after loading only 44GB (with 1G region size), so I don't think
> it's that rare. I'm also running with something like 30 handlers. My tests
> are with 80 concurrent clients.
>
> So consider me a -0 - if no one else runs into this bug we may as well
> release, but it seems a step backward in stability for me from the 20100621
> (the last one I did significant load testing on)
>
> -Todd
>
> On Fri, Sep 3, 2010 at 12:05 PM, Stack <st...@duboce.net> wrote:
>
>> +1
>>
>> Its running in production at SU.
>>
>> St.Ack
>>
>> On Mon, Aug 30, 2010 at 5:29 PM, Jean-Daniel Cryans <jd...@apache.org>
>> wrote:
>> > BTW I'm +1, I ran the unit tests, did cluster testing with YCSB, and
>> > deployed it in almost all our environments here (this will be
>> > completed before the voting period ends).
>> >
>> > J-D
>> >
>> > On Mon, Aug 30, 2010 at 4:39 PM, Jean-Daniel Cryans <jd...@apache.org>
>> wrote:
>> >> It's time for another DR since the new master code is about to be
>> >> merged in and we have a few fixes and improvements that'd beneficiate
>> >> from more exposure. I branched from trunk this morning and created a
>> >> new tag. (See http://wiki.apache.org/hadoop/Hbase/HBaseVersions for
>> >> more on what these 0.89.x releases are all about)
>> >>
>> >> Source binary and source tar balls are available here:
>> >>
>> >> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/
>> >>
>> >> You can also browse the candidate documentation here:
>> >>
>> >>
>> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/hbase-0.89.20100830/docs/
>> >>
>> >> Issues resolved since 0.89.20100726, our second 0.89.x release, are
>> >> roughly ~23 issues odd including fixed deadlocks, better handling of
>> >> IOEs during splits and improvements for filters: see
>> >> http://su.pr/2HwiUe
>> >>
>> >> Shall we release this candidate as the third in our 0.89.x series of
>> >> developer releases?
>> >>
>> >> Please see previous threads on 0.89 releases for more information
>> >> about the purpose of this release candidate - in particular, this
>> >> 'developer release' is for those who can tolerate risk and who are
>> >> willing to give feedback in advance of our next major release. We're
>> >> not making any guarantees that this is bug free. Its definitely not
>> >> for production deploys.
>> >>
>> >> We'll do another release like this in a few weeks after the new master
>> >> code has gone in.
>> >>
>> >> Please vote by Monday, September 6th.
>> >>
>> >> Thanks,
>> >> J-D
>> >>
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: [VOTE] Release 'development release' HBase 0.89.2010830 rc1?
Posted by Todd Lipcon <to...@cloudera.com>.
I did some load tests on this afternoon and ran into this bug:
https://issues.apache.org/jira/browse/HBASE-2964
I got this after loading only 44GB (with 1G region size), so I don't think
it's that rare. I'm also running with something like 30 handlers. My tests
are with 80 concurrent clients.
So consider me a -0 - if no one else runs into this bug we may as well
release, but it seems a step backward in stability for me from the 20100621
(the last one I did significant load testing on)
-Todd
On Fri, Sep 3, 2010 at 12:05 PM, Stack <st...@duboce.net> wrote:
> +1
>
> Its running in production at SU.
>
> St.Ack
>
> On Mon, Aug 30, 2010 at 5:29 PM, Jean-Daniel Cryans <jd...@apache.org>
> wrote:
> > BTW I'm +1, I ran the unit tests, did cluster testing with YCSB, and
> > deployed it in almost all our environments here (this will be
> > completed before the voting period ends).
> >
> > J-D
> >
> > On Mon, Aug 30, 2010 at 4:39 PM, Jean-Daniel Cryans <jd...@apache.org>
> wrote:
> >> It's time for another DR since the new master code is about to be
> >> merged in and we have a few fixes and improvements that'd beneficiate
> >> from more exposure. I branched from trunk this morning and created a
> >> new tag. (See http://wiki.apache.org/hadoop/Hbase/HBaseVersions for
> >> more on what these 0.89.x releases are all about)
> >>
> >> Source binary and source tar balls are available here:
> >>
> >> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/
> >>
> >> You can also browse the candidate documentation here:
> >>
> >>
> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/hbase-0.89.20100830/docs/
> >>
> >> Issues resolved since 0.89.20100726, our second 0.89.x release, are
> >> roughly ~23 issues odd including fixed deadlocks, better handling of
> >> IOEs during splits and improvements for filters: see
> >> http://su.pr/2HwiUe
> >>
> >> Shall we release this candidate as the third in our 0.89.x series of
> >> developer releases?
> >>
> >> Please see previous threads on 0.89 releases for more information
> >> about the purpose of this release candidate - in particular, this
> >> 'developer release' is for those who can tolerate risk and who are
> >> willing to give feedback in advance of our next major release. We're
> >> not making any guarantees that this is bug free. Its definitely not
> >> for production deploys.
> >>
> >> We'll do another release like this in a few weeks after the new master
> >> code has gone in.
> >>
> >> Please vote by Monday, September 6th.
> >>
> >> Thanks,
> >> J-D
> >>
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: [VOTE] Release 'development release' HBase 0.89.2010830 rc1?
Posted by Stack <st...@duboce.net>.
+1
Its running in production at SU.
St.Ack
On Mon, Aug 30, 2010 at 5:29 PM, Jean-Daniel Cryans <jd...@apache.org> wrote:
> BTW I'm +1, I ran the unit tests, did cluster testing with YCSB, and
> deployed it in almost all our environments here (this will be
> completed before the voting period ends).
>
> J-D
>
> On Mon, Aug 30, 2010 at 4:39 PM, Jean-Daniel Cryans <jd...@apache.org> wrote:
>> It's time for another DR since the new master code is about to be
>> merged in and we have a few fixes and improvements that'd beneficiate
>> from more exposure. I branched from trunk this morning and created a
>> new tag. (See http://wiki.apache.org/hadoop/Hbase/HBaseVersions for
>> more on what these 0.89.x releases are all about)
>>
>> Source binary and source tar balls are available here:
>>
>> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/
>>
>> You can also browse the candidate documentation here:
>>
>> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/hbase-0.89.20100830/docs/
>>
>> Issues resolved since 0.89.20100726, our second 0.89.x release, are
>> roughly ~23 issues odd including fixed deadlocks, better handling of
>> IOEs during splits and improvements for filters: see
>> http://su.pr/2HwiUe
>>
>> Shall we release this candidate as the third in our 0.89.x series of
>> developer releases?
>>
>> Please see previous threads on 0.89 releases for more information
>> about the purpose of this release candidate - in particular, this
>> 'developer release' is for those who can tolerate risk and who are
>> willing to give feedback in advance of our next major release. We're
>> not making any guarantees that this is bug free. Its definitely not
>> for production deploys.
>>
>> We'll do another release like this in a few weeks after the new master
>> code has gone in.
>>
>> Please vote by Monday, September 6th.
>>
>> Thanks,
>> J-D
>>
>
Re: [VOTE] Release 'development release' HBase 0.89.2010830 rc1?
Posted by Jean-Daniel Cryans <jd...@apache.org>.
BTW I'm +1, I ran the unit tests, did cluster testing with YCSB, and
deployed it in almost all our environments here (this will be
completed before the voting period ends).
J-D
On Mon, Aug 30, 2010 at 4:39 PM, Jean-Daniel Cryans <jd...@apache.org> wrote:
> It's time for another DR since the new master code is about to be
> merged in and we have a few fixes and improvements that'd beneficiate
> from more exposure. I branched from trunk this morning and created a
> new tag. (See http://wiki.apache.org/hadoop/Hbase/HBaseVersions for
> more on what these 0.89.x releases are all about)
>
> Source binary and source tar balls are available here:
>
> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/
>
> You can also browse the candidate documentation here:
>
> http://people.apache.org/~jdcryans/hbase-0.89.20100830-candidate-1/hbase-0.89.20100830/docs/
>
> Issues resolved since 0.89.20100726, our second 0.89.x release, are
> roughly ~23 issues odd including fixed deadlocks, better handling of
> IOEs during splits and improvements for filters: see
> http://su.pr/2HwiUe
>
> Shall we release this candidate as the third in our 0.89.x series of
> developer releases?
>
> Please see previous threads on 0.89 releases for more information
> about the purpose of this release candidate - in particular, this
> 'developer release' is for those who can tolerate risk and who are
> willing to give feedback in advance of our next major release. We're
> not making any guarantees that this is bug free. Its definitely not
> for production deploys.
>
> We'll do another release like this in a few weeks after the new master
> code has gone in.
>
> Please vote by Monday, September 6th.
>
> Thanks,
> J-D
>