old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Jonathan Gray <jg...@facebook.com>]

Deep within my soul I do not want to do this.

But it might be practical.

FB is going into prod w/ the old master and we've been doing work stabilizing it in ways that do not apply at all to the new one (especially around zk).  That'll make having two different active branches a bit of a nightmare but right now some of these patches are not even available as we've been operating under the assumption that the release would be on the new master (and the patches don't apply on trunk).

I guess SU is also putting an 0.89 old master release into prod.

What's a little unfortunate is that the 0.89 releases include the unnecessary move of some transition communication into ZK.  That was put in as a first step towards new master before we decided to branch it off.

The question for me would be whether we think it's at all feasible to get the new master 0.90 released in time for HW.  If not, maybe we should consider taking an 0.89 branch, making it an 0.90 branch, and focus on stabilizing for release in time for HW, as Todd suggests.

There are other ramifications of this.  It would be very difficult for me to get my flush/compact/split improvements in to the old master as the new implementation I've been working on relies completely on the new stuff.  But maybe better to punt that for 0.92 as well so can really nail it?

The other factor is the enormous amount of ZK improvements that were done in the new master branch.  It's a real fuckin mess in 0.89 releases, though I've done a bit of cleanup already towards making it at least tenable for production.

My main concern is that this move will push back any release of the new master significantly.  There are countless improvements in the codebase that came along with the rewrite, well beyond just zk transitions.  But doing a production release in time for HW is probably the most important thing.

JG

> -----Original Message-----
> From: Todd Lipcon [mailto:todd@cloudera.com]
> Sent: Tuesday, September 21, 2010 8:10 AM
> To: dev
> Subject: Re: Millions of photos into Hbase
> 
> On Mon, Sep 20, 2010 at 5:47 PM, Ryan Rawson <ry...@gmail.com>
> wrote:
> > Hey,
> >
> > There is actually only 1 active branch of hbase, that being the 0.89
> > release, which is based on 'trunk'.  We have snapshotted a series of
> > 0.89 "developer releases" in hopes that people would try them our and
> > start thinking about the next major version.  One of these is what SU
> > is running prod on.
> >
> > At this point tracking 0.89 and which ones are the 'best' peach sets
> > to run is a bit of a contact sport, but if you are serious about not
> > losing data it is worthwhile.  SU is based on the most recent DR with
> > a few minor patches of our own concoction brought in.  If current
> > works, but some Master ops are slow, and there are a few patches on
> > top of that.  I'll poke about and see if its possible to publish to a
> > github branch or something.
> >
> 
> This is why I kind of want to release the latest 0.89 dev release as
> 0.90, and push off the new master stuff as 0.92 :)
> 
> -Todd
> 
> --
> Todd Lipcon
> Software Engineer, Cloudera

RE: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Jonathan Gray <jg...@facebook.com>]

Yes, there are way more changes/improvements made beyond the initial scope of working master failover / zk-based region transitions.  Lots of cleanup, removal of many of the nasty bits we've always hated, and the groundwork is laid for a lot of future improvements.

I'm not apprehensive.  I think many of the points raised are valid, but my concerns have mostly been addressed at this point.  I'm going to be spending the next 10 days or so helping stabilize and test.  We'll see where we get by then.

JG
 
> -----Original Message-----
> From: Andrew Purtell [mailto:apurtell@apache.org]
> Sent: Thursday, September 23, 2010 3:47 AM
> To: dev@hbase.apache.org
> Subject: Re: old master to 0.90 and new master to 0.92? (was RE:
> Millions of photos into Hbase)
> 
> Thanks. I'm going to try it out right away.
> 
> > From: Ryan Rawson
> > I've been using the new master a bit, and the improvements are just
> > too dramatic to ignore or delay.  Instant table transitions, clusters
> > that actually shut down, etc, etc.
> >
> >
> > On Wed, Sep 22, 2010 at 3:11 PM, Andrew Purtell <ap...@apache.org>
> > wrote:
> > > If JG is apprehensive about putting the new master
> > into 0.90, that's good enough for me. But I'll vote +0. I'm
> > too unfamiliar with the details.
> 
> 
> 
> 

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Andrew Purtell <ap...@apache.org>]

Thanks. I'm going to try it out right away.

> From: Ryan Rawson
> I've been using the new master a bit, and the improvements are just
> too dramatic to ignore or delay.  Instant table transitions, clusters
> that actually shut down, etc, etc.
> 
>
> On Wed, Sep 22, 2010 at 3:11 PM, Andrew Purtell <ap...@apache.org>
> wrote:
> > If JG is apprehensive about putting the new master
> into 0.90, that's good enough for me. But I'll vote +0. I'm
> too unfamiliar with the details.



      

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Ryan Rawson <ry...@gmail.com>]

I've been using the new master a bit, and the improvements are just
too dramatic to ignore or delay.  Instant table transitions, clusters
that actually shut down, etc, etc.



On Wed, Sep 22, 2010 at 3:11 PM, Andrew Purtell <ap...@apache.org> wrote:
> If JG is apprehensive about putting the new master into 0.90, that's good enough for me. But I'll vote +0. I'm too unfamiliar with the details.
>
> Best regards,
>
>    - Andy
>
> Why is this email five sentences or less?
> http://five.sentenc.es/
>
>
>
>
>
>

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Andrew Purtell <ap...@apache.org>]

If JG is apprehensive about putting the new master into 0.90, that's good enough for me. But I'll vote +0. I'm too unfamiliar with the details. 

Best regards,

    - Andy

Why is this email five sentences or less?
http://five.sentenc.es/



      

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Todd Lipcon <to...@cloudera.com>]

On Wed, Sep 22, 2010 at 5:47 AM, Jonathan Gray <jg...@facebook.com> wrote:

> > > We also confuse the 0.90 'message' given we've been talking about new
> > > master at HUGs and here on the lists with a good while now.
> >
> > This might be a little naïve but what about calling the "SU"-build
> > 0.88 and release it (with a small disclaimer). It would make sense in
> > that the 0.89-dev releases have more stuff in them than 0.88 and it
> > would still allow for a 0.90 release with all the announced features.
>
> I like this too... if we go the direction of an old master release.
>

I'm -1 on an out-of-order release like this. I think it will really confuse
people, please won't work correctly with upgrades on packaging systems.

I know we've been promising "new master for 0.90" at meetups and stuff, but
outside the dev team, does anyone really care about the implementation
details? The feature here isn't a new master, it's more stable master
failover, better locality, etc. Whether we have that or say that it's going
to be in the next version, I don't think people will be too upset.

The argument that it's hell to maintain two branches is a fair one, but
unless we can get trunk usable very soon now, we're already going to be in
that situation. People are migrating production instances to 0.89.x now, and
once people go to production they are very hesitant to upgrade, regardless
of what we tell them.

-Todd

-- 
Todd Lipcon
Software Engineer, Cloudera

RE: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Jonathan Gray <jg...@facebook.com>]

> > We also confuse the 0.90 'message' given we've been talking about new
> > master at HUGs and here on the lists with a good while now.
> 
> This might be a little naïve but what about calling the "SU"-build
> 0.88 and release it (with a small disclaimer). It would make sense in
> that the 0.89-dev releases have more stuff in them than 0.88 and it
> would still allow for a 0.90 release with all the announced features.

I like this too... if we go the direction of an old master release.

It addresses most of the issues, besides slowing release of new master.  So not necessarily +1 on doing it but this would be better than an old master 0.90 release imo.

Good idea Lars.

JG

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Stack <st...@duboce.net>]

On Tue, Sep 21, 2010 at 4:54 PM, Lars Francke <la...@gmail.com> wrote:
>> We also confuse the 0.90 'message' given we've been talking about new
>> master at HUGs and here on the lists with a good while now.
>
> This might be a little naïve but what about calling the "SU"-build
> 0.88 and release it (with a small disclaimer). It would make sense in
> that the 0.89-dev releases have more stuff in them than 0.88 and it
> would still allow for a 0.90 release with all the announced features.
>

Thats an idea.

> As a side note: I plan to finish work on the Maven changes in time for
> HW but tests keep on failing (Hudson has the same problem...) which
> doesn't make it easy to completely test the build.

Sorry about that.  I've been picking away at them from time to time
but some require more than a glance to fix (For me at least, most of
what fails on hudson does not fail for me here below).

I don't know if
> I'll be able to get Thrift done though, sorry. I'm doing this in my
> spare time and I didn't have a lot of that :( but with a bit of luck I
> can work on HBase in my new job. But I'll upload updated Thrift files
> in the next few days if someone wants to check what I've done (modeled
> it after Jeff's Avro work).
>

Thanks for the update and good luck in the new Job.

St.Ack

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Lars Francke <la...@gmail.com>]

> We also confuse the 0.90 'message' given we've been talking about new
> master at HUGs and here on the lists with a good while now.

This might be a little naïve but what about calling the "SU"-build
0.88 and release it (with a small disclaimer). It would make sense in
that the 0.89-dev releases have more stuff in them than 0.88 and it
would still allow for a 0.90 release with all the announced features.

As a side note: I plan to finish work on the Maven changes in time for
HW but tests keep on failing (Hudson has the same problem...) which
doesn't make it easy to completely test the build. I don't know if
I'll be able to get Thrift done though, sorry. I'm doing this in my
spare time and I didn't have a lot of that :( but with a bit of luck I
can work on HBase in my new job. But I'll upload updated Thrift files
in the next few days if someone wants to check what I've done (modeled
it after Jeff's Avro work).

Cheers,
Lars

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Stack <st...@duboce.net>]

On Tue, Sep 21, 2010 at 10:19 AM, Todd Lipcon <to...@cloudera.com> wrote:
> CDH3b3 will be ready for Hadoop world, and we'd kind of like to freeze
> component versions at this point in the beta cycle. So if 0.90 is out,
> that would be great. We can certainly work with what we've got
> (20100830 minus ZK assignment) but if a production-worth new master
> isn't ready by the end of October or so we'll probably push that out
> til CDH4.
>

Ok.

Its my thought that we'd have a production ready master way before end
of October.


>> I'm thinking we can afford to take such a position
>> because if someone wants durable hbase now, they can run with the
>> SU-prod 0.89.x that J-D is about to put up.
>>
>
> This is going to be an official Apache release, right? I guess my
> question is: if it's stable and usable for production, why call it a
> "development release"? If we're recommending it for all new users over
> and above 0.20.6, then it seems like this should be deemed stable (ie
> even release number).
>

I was not talking of recommending the next 0.89.x to all new users
over 0.20.6.  I thought it plain that running a release in production
at SU did not mean the release 'stable and useable' by others. SU has
3 hbase committers aboard to fix and hand-hold the software over rough
patches.

I was thinking that the next 0.89.x comes w/ the same caveats as
previous 0.89.xs, that its a 'developer release' for those willing to
put up w/ some rough edges and that its just another step on the way
to 0.90.0.


>> Putting off new master till 0.92 means it'll be maybe 6 months before
>> it appears.  During this time we'll be paying a high price keeping up
>> two disparate branches -- TRUNK w/ new master and the release branch
>> -- shoe-horning patches to fit both.
>>
>
> If you guys are running 20100830 in production, won't you be doing
> that anyway? Assumedly we'd treat this 0.90 as "no new features" and
> put the new features into 0.91.x leading up to 0.92?
>

Nope.  We'd move to new master release.  If it don't work for us, we'd
feel a little awkward recommending it to others.


>> We also confuse the 0.90 'message' given we've been talking about new
>> master at HUGs and here on the lists with a good while now.
>
> True. The question is whether we prefer to slip time or slip scope. In
> my opinion slipping scope is better - it's open source and people
> understand that schedules slip. Keeping strong release momentum up
> helps adoption and will get people off 0.20.6 which no one really
> wants to support anymore.
>

This is the question.  I'm just suggesting that new master MAY not be
that far out.  I want to do another couple of days work on it and then
have us make a call; i.e. vote that we press on to get new master into
0.90 or punt on new master for 0.90.

St.Ack

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Todd Lipcon <to...@cloudera.com>]

On Tue, Sep 21, 2010 at 10:11 AM, Stack <st...@duboce.net> wrote:
> On Tue, Sep 21, 2010 at 4:49 AM, Jonathan Gray <jg...@facebook.com> wrote:
>> My main concern is that this move will push back any release of the new master significantly.  There are countless improvements in the codebase that came along with the rewrite, well beyond just zk transitions.  But doing a production release in time for HW is probably the most important thing.
>>
>
> IMO, we're almost there w/ the new master.  I'd like to wait till
> Thursday or Friday before making a call.
>
> Regards HW, we might consider pressing on with new master even if it
> means we don't exactly line up a new master 0.89.x (or a 0.90.xRC)
> release w/ HW.  You fellas talking at HW can fudge it some if it comes
> to that (We won't be the only ones.  CDH3 will not be ready for HW
> apparently).

CDH3b3 will be ready for Hadoop world, and we'd kind of like to freeze
component versions at this point in the beta cycle. So if 0.90 is out,
that would be great. We can certainly work with what we've got
(20100830 minus ZK assignment) but if a production-worth new master
isn't ready by the end of October or so we'll probably push that out
til CDH4.

> I'm thinking we can afford to take such a position
> because if someone wants durable hbase now, they can run with the
> SU-prod 0.89.x that J-D is about to put up.
>

This is going to be an official Apache release, right? I guess my
question is: if it's stable and usable for production, why call it a
"development release"? If we're recommending it for all new users over
and above 0.20.6, then it seems like this should be deemed stable (ie
even release number).

> Putting off new master till 0.92 means it'll be maybe 6 months before
> it appears.  During this time we'll be paying a high price keeping up
> two disparate branches -- TRUNK w/ new master and the release branch
> -- shoe-horning patches to fit both.
>

If you guys are running 20100830 in production, won't you be doing
that anyway? Assumedly we'd treat this 0.90 as "no new features" and
put the new features into 0.91.x leading up to 0.92?

> We also confuse the 0.90 'message' given we've been talking about new
> master at HUGs and here on the lists with a good while now.

True. The question is whether we prefer to slip time or slip scope. In
my opinion slipping scope is better - it's open source and people
understand that schedules slip. Keeping strong release momentum up
helps adoption and will get people off 0.20.6 which no one really
wants to support anymore.

-Todd

-- 
Todd Lipcon
Software Engineer, Cloudera

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Stack <st...@duboce.net>]

On Tue, Sep 21, 2010 at 4:49 AM, Jonathan Gray <jg...@facebook.com> wrote:
> My main concern is that this move will push back any release of the new master significantly.  There are countless improvements in the codebase that came along with the rewrite, well beyond just zk transitions.  But doing a production release in time for HW is probably the most important thing.
>

IMO, we're almost there w/ the new master.  I'd like to wait till
Thursday or Friday before making a call.

Regards HW, we might consider pressing on with new master even if it
means we don't exactly line up a new master 0.89.x (or a 0.90.xRC)
release w/ HW.  You fellas talking at HW can fudge it some if it comes
to that (We won't be the only ones.  CDH3 will not be ready for HW
apparently).  I'm thinking we can afford to take such a position
because if someone wants durable hbase now, they can run with the
SU-prod 0.89.x that J-D is about to put up.

Putting off new master till 0.92 means it'll be maybe 6 months before
it appears.  During this time we'll be paying a high price keeping up
two disparate branches -- TRUNK w/ new master and the release branch
-- shoe-horning patches to fit both.

We also confuse the 0.90 'message' given we've been talking about new
master at HUGs and here on the lists with a good while now.

St.Ack

Re: old master to 0.90 and new master to 0.92? (was RE: Millions of photos into Hbase)

[Jean-Daniel Cryans <jd...@apache.org>]

So while you were in the plane a few things were discussed :)

In the thread "[VOTE] Release 'development release' HBase 0.89.2010830
rc2", it was decided that we sink RC2 in favor of a new RC with a
rollback of the zk-based assignment. This is what we're running here,
and Ryan published our repo yesterday
http://github.com/stumbleupon/hbase  (see the top of the CHANGES file
for what we added)

Also some of your arguments correlate those of Todd, expressed in the
thread "Next release" that was posted on 09/15. Stack and Andrew
expressed their opinions in favor of Todd's option #1 (although Stack
said that he would decide tomorrow if he changes his opinion after
some more work on the master).

My personal opinion leans towards Todd's option #2. 0.20.0 is now more
than a year old, we are able to release almost production-ready code
that is durable, so why not move forward with it (this is what we did
here). There are a few issues to fix, but it's small compared to
getting the new master in shape plus getting the rest working (like
the replication code's interaction with ZooKeeper that needs to be
redone).

J-D

On Tue, Sep 21, 2010 at 4:49 AM, Jonathan Gray <jg...@facebook.com> wrote:
> Deep within my soul I do not want to do this.
>
> But it might be practical.
>
> FB is going into prod w/ the old master and we've been doing work stabilizing it in ways that do not apply at all to the new one (especially around zk).  That'll make having two different active branches a bit of a nightmare but right now some of these patches are not even available as we've been operating under the assumption that the release would be on the new master (and the patches don't apply on trunk).
>
> I guess SU is also putting an 0.89 old master release into prod.
>
> What's a little unfortunate is that the 0.89 releases include the unnecessary move of some transition communication into ZK.  That was put in as a first step towards new master before we decided to branch it off.
>
> The question for me would be whether we think it's at all feasible to get the new master 0.90 released in time for HW.  If not, maybe we should consider taking an 0.89 branch, making it an 0.90 branch, and focus on stabilizing for release in time for HW, as Todd suggests.
>
> There are other ramifications of this.  It would be very difficult for me to get my flush/compact/split improvements in to the old master as the new implementation I've been working on relies completely on the new stuff.  But maybe better to punt that for 0.92 as well so can really nail it?
>
> The other factor is the enormous amount of ZK improvements that were done in the new master branch.  It's a real fuckin mess in 0.89 releases, though I've done a bit of cleanup already towards making it at least tenable for production.
>
> My main concern is that this move will push back any release of the new master significantly.  There are countless improvements in the codebase that came along with the rewrite, well beyond just zk transitions.  But doing a production release in time for HW is probably the most important thing.
>
> JG
>
>> -----Original Message-----
>> From: Todd Lipcon [mailto:todd@cloudera.com]
>> Sent: Tuesday, September 21, 2010 8:10 AM
>> To: dev
>> Subject: Re: Millions of photos into Hbase
>>
>> On Mon, Sep 20, 2010 at 5:47 PM, Ryan Rawson <ry...@gmail.com>
>> wrote:
>> > Hey,
>> >
>> > There is actually only 1 active branch of hbase, that being the 0.89
>> > release, which is based on 'trunk'.  We have snapshotted a series of
>> > 0.89 "developer releases" in hopes that people would try them our and
>> > start thinking about the next major version.  One of these is what SU
>> > is running prod on.
>> >
>> > At this point tracking 0.89 and which ones are the 'best' peach sets
>> > to run is a bit of a contact sport, but if you are serious about not
>> > losing data it is worthwhile.  SU is based on the most recent DR with
>> > a few minor patches of our own concoction brought in.  If current
>> > works, but some Master ops are slow, and there are a few patches on
>> > top of that.  I'll poke about and see if its possible to publish to a
>> > github branch or something.
>> >
>>
>> This is why I kind of want to release the latest 0.89 dev release as
>> 0.90, and push off the new master stuff as 0.92 :)
>>
>> -Todd
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>