You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Roman Shaposhnik <rv...@apache.org> on 2011/11/16 22:54:56 UTC
Now that 0.2.0 is out...
...it is time to start thinking about 0.3.0 (and beyond!) ;-)
I will list some ideas I have in this email thread and if there's
traction, we can put a few of them up to a VOTE.
* Bigtop release model
1. I really would like Bigtop to have a quarterly release schedule.
It is true that date-driven release are frowned upon by Apache
in general, but I think this model works well for Bigtop.
2. For any project that follows a fixed date-driven schedule the
versioning model that Ubuntu has come up with makes the most
sense as far as I'm concerned (year.month). I've also grown fond
of given releases names (e.g. "Lucid Lynx"). Perhaps we can name
Bigtop releases by Greek/Roman gods. Or may be it should be animals
(since it is Bigtop after all!) -- chime in if you have a good suggestion
3. On a more serious note, I would like to propose that we reserve
backporting
activity (and any chance of releasing from an already released branches) to
dealing with catastrophic events. Things like security breaches,
etc. Basically, no
backports unless absolutely necessary.
* Bigtop and Hadoop 0.23/0.22-based distributions
1. Given the uncertainty around release of downstream components compatible
with the MR2 APIs it is highly unlikely that the next *release* of
Bigtop will be
in a position to use 0.23/0.22.
2. That said, we will continue working on those releases in two
long lived branches
hadoop-0.23/hadoop-0.22. So far we haven't had any clear policy on how to
handle commits going into those branches. So far I'm aware of 3
major things that
we need to agree upon (chime in, if there's more):
* how frequently to rebase them on trunk
>>> I propose weekly
* what's the JIRA version for filing issues against them
>>> I propose esignated versions hadoop-0.23/hadoop-0.22
* what's the review/commit policy
>>> I propose the usual one we have for Bigtop:
file a JIRA, get a +1
* Bigtop as a place for integration testing of RCs
1. We are getting a reasonable traction with downstream
communities turning to
Bigtop for integration testing of their RCs. It would be nice to
have a formal policy
on how we can accommodate these requests. So far, I've been using a fork of
Bigtop on Github to do ad-hoc builds and validation, but perhaps
we can have
long-lived branch for that, or may be there's a better option. Let
me know what
do you think.
Thanks,
Roman.
Re: Now that 0.2.0 is out...
Posted by Roman Shaposhnik <rv...@apache.org>.
Let me try to see if I can summarize the discussion on this thread
so far:
On Wed, Nov 16, 2011 at 1:54 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> 1. Bigtop quarterly release schedule.
General consensus that this is a "good thing", but for now we might need to
turn around faster. Feels like punting it for at leas Bigtop 0.3.0 is the right
think to do, but lets revisit in 2012.
> 2. Code names
No Roman gods -- otherwise anything goes. Again, punting for now.
> 3. Limited backporting activity
Feels like we've agreed to that.
> 4. Bigtop 0.3.0
Is going to be the last one based on 20.20X branch (see a separate vote on that)
> 5. Bigtop based on .23 rebases
On-demand
> 6. Bigtop based on .23 JIRAs.
Every change has to have a JIRA, please do NOT set fixedVersion/affectedVersion.
> 7. Bigtop as a place for integration testing of RCs
RCs branch has been created for that purpose, with lax commit requirements.
Thanks and this thread is now officially closed ;-)
Roman.
Re: Now that 0.2.0 is out...
Posted by Bruno Mahé <bm...@apache.org>.
On 11/16/2011 10:27 PM, Bruno Mahé wrote:
> 1. I would actually be rather in favour of a faster release cycle. But a
> quarterly one is probably a good middle ground. This assume we don't get
> strict about it. For instance an important update of one of the
> component may warrant a new release.
>
> 2. I only care about monotonically increasing numbers. But codenames are
> fun :)
>
> 3. It depends on how we are planning releases (hence your email).
> I only care about a stable release/branch one could trust to maintain a
> production grade cluster and another one where we could go forward and
> experiment. I was under the impression we would be heading toward a
> 0.3.0 based on a next gen hadoop, which would definitely not be
> production ready.
> So if the consensus is not to update hadoop version for the next bigtop
> release, backporting patches to the 0.2.0 branch does not make sense.
Of course I meant to a "next gen" version such as 0.22 or 0.23.
> But if we plan to update hadoop to a next gen version, I will definitely
> maintain and backport patches to the 0.2.0 branch. But this is
> implementation details.
>
Re: Now that 0.2.0 is out...
Posted by Bruno Mahé <bm...@apache.org>.
1. I would actually be rather in favour of a faster release cycle. But a
quarterly one is probably a good middle ground. This assume we don't get
strict about it. For instance an important update of one of the
component may warrant a new release.
2. I only care about monotonically increasing numbers. But codenames are
fun :)
3. It depends on how we are planning releases (hence your email).
I only care about a stable release/branch one could trust to maintain a
production grade cluster and another one where we could go forward and
experiment. I was under the impression we would be heading toward a
0.3.0 based on a next gen hadoop, which would definitely not be
production ready.
So if the consensus is not to update hadoop version for the next bigtop
release, backporting patches to the 0.2.0 branch does not make sense.
But if we plan to update hadoop to a next gen version, I will definitely
maintain and backport patches to the 0.2.0 branch. But this is
implementation details.
4. I agree completely. Although I would like also to make sure that any
patch we apply on a branch should have an upstream JIRA as a
filename/identifier so anyone can follow up on any patch and we can
ensure they all get commited upstream by the time a new upstream version
gets released.
Note that I used the term "should" and not "must" in order to make it
more of a guideline instead of a constraint.
5. Please define requests.
It seems to be more of an infrastructure issue.
I am not sure this warrant any specific process or branch to do such
thing. Although we could always help making things easier by documenting
more the build and test processes in Bigtop as well as maybe trying to
provide some parametrized jobs on our jenkins instance (where parameters
could be specific patches).
On 11/16/2011 01:54 PM, Roman Shaposhnik wrote:
> ...it is time to start thinking about 0.3.0 (and beyond!) ;-)
>
> I will list some ideas I have in this email thread and if there's
> traction, we can put a few of them up to a VOTE.
>
> * Bigtop release model
>
> 1. I really would like Bigtop to have a quarterly release schedule.
> It is true that date-driven release are frowned upon by Apache
> in general, but I think this model works well for Bigtop.
>
> 2. For any project that follows a fixed date-driven schedule the
> versioning model that Ubuntu has come up with makes the most
> sense as far as I'm concerned (year.month). I've also grown fond
> of given releases names (e.g. "Lucid Lynx"). Perhaps we can name
> Bigtop releases by Greek/Roman gods. Or may be it should be animals
> (since it is Bigtop after all!) -- chime in if you have a good suggestion
>
> 3. On a more serious note, I would like to propose that we reserve
> backporting
> activity (and any chance of releasing from an already released branches) to
> dealing with catastrophic events. Things like security breaches,
> etc. Basically, no
> backports unless absolutely necessary.
>
> * Bigtop and Hadoop 0.23/0.22-based distributions
>
> 1. Given the uncertainty around release of downstream components compatible
> with the MR2 APIs it is highly unlikely that the next *release* of
> Bigtop will be
> in a position to use 0.23/0.22.
>
> 2. That said, we will continue working on those releases in two
> long lived branches
> hadoop-0.23/hadoop-0.22. So far we haven't had any clear policy on how to
> handle commits going into those branches. So far I'm aware of 3
> major things that
> we need to agree upon (chime in, if there's more):
> * how frequently to rebase them on trunk
> >>> I propose weekly
> * what's the JIRA version for filing issues against them
> >>> I propose esignated versions hadoop-0.23/hadoop-0.22
> * what's the review/commit policy
> >>> I propose the usual one we have for Bigtop:
> file a JIRA, get a +1
>
> * Bigtop as a place for integration testing of RCs
>
> 1. We are getting a reasonable traction with downstream
> communities turning to
> Bigtop for integration testing of their RCs. It would be nice to
> have a formal policy
> on how we can accommodate these requests. So far, I've been using a fork of
> Bigtop on Github to do ad-hoc builds and validation, but perhaps
> we can have
> long-lived branch for that, or may be there's a better option. Let
> me know what
> do you think.
>
> Thanks,
> Roman.
Re: Now that 0.2.0 is out...
Posted by Roman Shaposhnik <rv...@apache.org>.
On Thu, Nov 17, 2011 at 6:02 PM, Konstantin Boudnik <co...@apache.org> wrote:
> As for the question where to host such a branch, I'd say let's have next to
> the development and release branches of BigTop and call self-explanatory. Say,
> wasteland or something to that effect. So, whoever need can check it out and
> experiment in any way.
I'm about to create a branch called RCs for that purpose. The only
work that is ever
expected to be done there is to validate upstream. No regular work
will be performed
on that branch and its history is only useful as points in time, not
as a continuous
stream.
I guess what I'm saying is -- don't mind that branch ;-)
Thanks,
Roman.
Re: Now that 0.2.0 is out...
Posted by Konstantin Boudnik <co...@apache.org>.
On Thu, Nov 17, 2011 at 02:55PM, Roman Shaposhnik wrote:
> On Wed, Nov 16, 2011 at 11:12 PM, Konstantin Boudnik <co...@apache.org> wrote:
> >> ═ ═ 1. We are getting a reasonable traction with downstream
> >> communities turning to
> >> ═ ═ Bigtop for integration testing of their RCs. It would be nice to
> >> have a formal policy
> >> ═ ═ on how we can accommodate these requests. So far, I've been using a fork of
> >> ═ ═ Bigtop on Github to do ad-hoc builds and validation, but perhaps
> >> we can have
> >> ═ ═ long-lived branch for that, or may be there's a better option. Let
> >> me know what
> >> ═ ═ do you think.
> >
> > can we do a parametrized stack where we can set the versions of included
> > components separately from BigTop's pom and read them at the runtime? ═I guess
> > Gradle would solve this problem for us but for now we might have to go with
> > some inlined Groovy hacks.
>
> Here's the problem we're trying to solve here: we are starting to get requests
> for (at least!) validating RCs of the projects that constitute Bigtop
> so that they
> know there's no regression compared to what was there. Here's a practical
> example -- ZK RC 3.3.4 has been posted. I would like to verify that replacing
> a single component (ZK) in a Bigtop 0.2.0 stack will NOT lead to a regression.
> Right now, I'm doing this in a rather ad-hoc manner pushing to my own GitHub
> repo and building (and soon testing) over here:
> http://bigtop01.cloudera.org:8080/view/RCs/?
>
> This works, but I'm not sure this is the best way to handle it.
As per our offline chat you explained to me that what I was proposing is
pretty much implemented (except for the automation part) and in fact the
branch above is serving exactly this purpose.
As for the question where to host such a branch, I'd say let's have next to
the development and release branches of BigTop and call self-explanatory. Say,
wasteland or something to that effect. So, whoever need can check it out and
experiment in any way.
Cos
Re: Now that 0.2.0 is out...
Posted by Roman Shaposhnik <rv...@apache.org>.
On Wed, Nov 16, 2011 at 11:12 PM, Konstantin Boudnik <co...@apache.org> wrote:
>> 1. We are getting a reasonable traction with downstream
>> communities turning to
>> Bigtop for integration testing of their RCs. It would be nice to
>> have a formal policy
>> on how we can accommodate these requests. So far, I've been using a fork of
>> Bigtop on Github to do ad-hoc builds and validation, but perhaps
>> we can have
>> long-lived branch for that, or may be there's a better option. Let
>> me know what
>> do you think.
>
> can we do a parametrized stack where we can set the versions of included
> components separately from BigTop's pom and read them at the runtime? I guess
> Gradle would solve this problem for us but for now we might have to go with
> some inlined Groovy hacks.
Here's the problem we're trying to solve here: we are starting to get requests
for (at least!) validating RCs of the projects that constitute Bigtop
so that they
know there's no regression compared to what was there. Here's a practical
example -- ZK RC 3.3.4 has been posted. I would like to verify that replacing
a single component (ZK) in a Bigtop 0.2.0 stack will NOT lead to a regression.
Right now, I'm doing this in a rather ad-hoc manner pushing to my own GitHub
repo and building (and soon testing) over here:
http://bigtop01.cloudera.org:8080/view/RCs/?
This works, but I'm not sure this is the best way to handle it.
Thanks,
Roman.
Re: Now that 0.2.0 is out...
Posted by Konstantin Boudnik <co...@apache.org>.
1. date-driven realease schedule is a double-edge sword, really.
What if time for a release came but there's no significant changes in the
upstream component worthy adding? Are we then switch to one-off
feature-driven schedule and postpone one at hands?
2. Ubuntu's +04 and +10 versions have different maintanance model behind.
Unless you are proposing something similar I don't a real compelling reason
to do that. It's gonna be confusing IMO, however I don't feel strongly one
way or another if you guys want to be fancy like that ;)
As for code names: I never been able to say what the hell is Lucis and
which one is Merkat. However, I know that I am running 10.04 on most of my
hosts and 11.04 of a laptop. And this is information enough, in my opinion.
I'd say having internal code names are fun - making them public is - again
- simply confusing.
Also, Greece is a place which now has a bad stench of "Democracy: the god
which failed" and with all due respect Roman - not everyone believe in your gods :D
and more below...
On Wed, Nov 16, 2011 at 01:54PM, Roman Shaposhnik wrote:
> ...it is time to start thinking about 0.3.0 (and beyond!) ;-)
>
> I will list some ideas I have in this email thread and if there's
> traction, we can put a few of them up to a VOTE.
>
> * Bigtop release model
>
> 1. I really would like Bigtop to have a quarterly release schedule.
> It is true that date-driven release are frowned upon by Apache
> in general, but I think this model works well for Bigtop.
>
> 2. For any project that follows a fixed date-driven schedule the
> versioning model that Ubuntu has come up with makes the most
> sense as far as I'm concerned (year.month). I've also grown fond
> of given releases names (e.g. "Lucid Lynx"). Perhaps we can name
> Bigtop releases by Greek/Roman gods. Or may be it should be animals
> (since it is Bigtop after all!) -- chime in if you have a good suggestion
>
> 3. On a more serious note, I would like to propose that we reserve
> backporting
> activity (and any chance of releasing from an already released branches) to
> dealing with catastrophic events. Things like security breaches,
> etc. Basically, no
> backports unless absolutely necessary.
3. +1 on that: backport are for critical fixes only and only as a thing of last resort
> * Bigtop and Hadoop 0.23/0.22-based distributions
>
> 1. Given the uncertainty around release of downstream components compatible
> with the MR2 APIs it is highly unlikely that the next *release* of
> Bigtop will be
> in a position to use 0.23/0.22.
>
> 2. That said, we will continue working on those releases in two
> long lived branches
> hadoop-0.23/hadoop-0.22. So far we haven't had any clear policy on how to
> handle commits going into those branches. So far I'm aware of 3
> major things that
> we need to agree upon (chime in, if there's more):
> * how frequently to rebase them on trunk
> >>> I propose weekly
I'd say it doesn't really matter and the rebases should be happening whenever
one commits to these branches, really.
> * what's the JIRA version for filing issues against them
> >>> I propose esignated versions hadoop-0.23/hadoop-0.22
> * what's the review/commit policy
> >>> I propose the usual one we have for Bigtop:
> file a JIRA, get a +1
Yup, makes sense.
> * Bigtop as a place for integration testing of RCs
>
> 1. We are getting a reasonable traction with downstream
> communities turning to
> Bigtop for integration testing of their RCs. It would be nice to
> have a formal policy
> on how we can accommodate these requests. So far, I've been using a fork of
> Bigtop on Github to do ad-hoc builds and validation, but perhaps
> we can have
> long-lived branch for that, or may be there's a better option. Let
> me know what
> do you think.
can we do a parametrized stack where we can set the versions of included
components separately from BigTop's pom and read them at the runtime? I guess
Gradle would solve this problem for us but for now we might have to go with
some inlined Groovy hacks.
Cos
> Thanks,
> Roman.
Re: Now that 0.2.0 is out...
Posted by Peter Linnell <pl...@apache.org>.
Thanks for organizing this clearly.
On 11/16/2011 01:54 PM, Roman Shaposhnik wrote:
> ...it is time to start thinking about 0.3.0 (and beyond!) ;-)
>
> I will list some ideas I have in this email thread and if there's
> traction, we can put a few of them up to a VOTE.
>
> * Bigtop release model
>
> 1. I really would like Bigtop to have a quarterly release schedule.
> It is true that date-driven release are frowned upon by Apache
> in general, but I think this model works well for Bigtop.
Strong +1 there. In other projects I have worked on, experience shows
this generally is a good thing. It avoids feature creep and keeps
development cycles well focused on achievable goals.
> 2. For any project that follows a fixed date-driven schedule the
> versioning model that Ubuntu has come up with makes the most
> sense as far as I'm concerned (year.month). I've also grown fond
> of given releases names (e.g. "Lucid Lynx"). Perhaps we can name
> Bigtop releases by Greek/Roman gods. Or may be it should be animals
> (since it is Bigtop after all!) -- chime in if you have a good suggestion
>
> 3. On a more serious note, I would like to propose that we reserve
> backporting
> activity (and any chance of releasing from an already released branches) to
> dealing with catastrophic events. Things like security breaches,
> etc. Basically, no
> backports unless absolutely necessary.
Sounds good to me.
>
> * Bigtop and Hadoop 0.23/0.22-based distributions
>
> 1. Given the uncertainty around release of downstream components compatible
> with the MR2 APIs it is highly unlikely that the next *release* of
> Bigtop will be
> in a position to use 0.23/0.22.
>
> 2. That said, we will continue working on those releases in two
> long lived branches
> hadoop-0.23/hadoop-0.22. So far we haven't had any clear policy on how to
> handle commits going into those branches. So far I'm aware of 3
> major things that
> we need to agree upon (chime in, if there's more):
> * how frequently to rebase them on trunk
> >>> I propose weekly
> * what's the JIRA version for filing issues against them
> >>> I propose esignated versions hadoop-0.23/hadoop-0.22
> * what's the review/commit policy
> >>> I propose the usual one we have for Bigtop:
> file a JIRA, get a +1
Also fine with me.
>
> * Bigtop as a place for integration testing of RCs
>
> 1. We are getting a reasonable traction with downstream
> communities turning to
> Bigtop for integration testing of their RCs. It would be nice to
> have a formal policy
> on how we can accommodate these requests. So far, I've been using a fork of
> Bigtop on Github to do ad-hoc builds and validation, but perhaps
> we can have
> long-lived branch for that, or may be there's a better option. Let
> me know what
> do you think.
>
> Thanks,
> Roman.
A long term branch would be a good start.
Cheers,
Peter