Release Criteria

[David Nalley <da...@gnsa.us>]

Hi folks,

One of the things I've been pondering of late is a set of release
criteria. E.g. here is what CloudStack MUST do to be considered for
release.

So as background there is a somewhat complex social contract that I
think we informally enter with our users. People expect us to release
when we project - and we (IMO) damage the credibility of the project
by suggesting a date and then missing, sometimes by a wide margin. In
reading some of the documentation that other folks have around
Time-based releases, I really like one of the analogies that Ubuntu
uses [1] which is that of a play in a theatre. Things may not be
perfect, people may not absolutely know all of their lines, but
tickets have been sold, the audience is standing outside waiting to
come in, and it's a pretty drastic event for the show not to go on.

At the same time, bringing multiple RCs forward which get voted down
doesn't necessarily inspire confidence. Some of this is caused by our
lack of comprehensive testing. I think many people have an innate
sense of what they think CloudStack should deliver in a product
release; but we also have a number of people who suggest that they are
timid in signing off on a product release simply because of the scope.
I think we should actively be setting the expectations of consumers of
our product. Particularly in the short term when we don't have
completely comprehensive testing, having a standard that we can test
to that is our 'minimum acceptable' makes sense in my mind.

I am going to spend a few days drafting a set of criteria, and I'll
post it to the wiki, and ask for feedback and help in refining it,
just wanted to give a heads up on what I am thinking, and hopefully
get some consensus around it being a worthwhile thing.

--David





[1] https://wiki.ubuntu.com/TimeBasedReleases

Re: Release Criteria

[David Nalley <da...@gnsa.us>]

On Tue, Sep 10, 2013 at 8:46 PM, David Nalley <da...@gnsa.us> wrote:
> Hi folks,
>
> One of the things I've been pondering of late is a set of release
> criteria. E.g. here is what CloudStack MUST do to be considered for
> release.
>

I've spent a lot of time thinking about this - and have modified my
opinion somewhat. I think focusing on the act of releasing as a point
for quality is myopic. I think we should be establishing the quality
that we demand at all times in our release and master branch - and we
should actively be verifying these things before they hit those
branches.

Not enforcing that quality up front leaves us in a place to always
play catch up. So I've changed the title of this to 'Release Branch
Criteria'; and here is my first draft:

https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+Branch+Criteria

Obviously this is a draft, and needs work - and probably needs more
people actually defining what is important. I did this off of the top
of my head, and there are plenty of other functionality that remains
to be tested.  This document also probably needs  rearranging to
reflect where/when things should be tested. (I am looking at you
Prasanna :) )

Thoughts, comments, flames?

--David

RE: Release Criteria

[Animesh Chaturvedi <an...@citrix.com>]

> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Tuesday, September 10, 2013 5:47 PM
> To: dev@cloudstack.apache.org
> Subject: Release Criteria
> 
> Hi folks,
> 
> One of the things I've been pondering of late is a set of release
> criteria. E.g. here is what CloudStack MUST do to be considered for
> release.
> 
> So as background there is a somewhat complex social contract that I
> think we informally enter with our users. People expect us to release
> when we project - and we (IMO) damage the credibility of the project by
> suggesting a date and then missing, sometimes by a wide margin. In
> reading some of the documentation that other folks have around Time-
> based releases, I really like one of the analogies that Ubuntu uses [1]
> which is that of a play in a theatre. Things may not be perfect, people
> may not absolutely know all of their lines, but tickets have been sold,
> the audience is standing outside waiting to come in, and it's a pretty
> drastic event for the show not to go on.
> 
> At the same time, bringing multiple RCs forward which get voted down
> doesn't necessarily inspire confidence. Some of this is caused by our
> lack of comprehensive testing. I think many people have an innate sense
> of what they think CloudStack should deliver in a product release; but
> we also have a number of people who suggest that they are timid in
> signing off on a product release simply because of the scope.
> I think we should actively be setting the expectations of consumers of
> our product. Particularly in the short term when we don't have
> completely comprehensive testing, having a standard that we can test to
> that is our 'minimum acceptable' makes sense in my mind.
> 
> I am going to spend a few days drafting a set of criteria, and I'll post
> it to the wiki, and ask for feedback and help in refining it, just
> wanted to give a heads up on what I am thinking, and hopefully get some
> consensus around it being a worthwhile thing.
> 
> --David
> 
> 
> 
[Animesh>] The thread has gone off topic. I am +1 on having a well-defined release criterion that is agreed upon. David looking forward to your draft and ready to help shape it
> 
> 
> [1] https://wiki.ubuntu.com/TimeBasedReleases

RE: Release Criteria

[Donal Lafferty <do...@citrix.com>]

> -----Original Message-----
> From: Prasanna Santhanam [mailto:tsp@apache.org]
> Sent: 11 September 2013 05:52
> To: dev@cloudstack.apache.org
> Subject: Re: Release Criteria
> 
> To get beta quality builds we need to absolutely treat the master branch as
> 'stable'. Never hurt it, automate against it, branch off quality builds from it
> and create packages and mirror them. That'll save us a ton of effort.
> 

+1

Problems with Master have greatly affected my ability to respond sympathetically to reviewer comments.  

Adding a new hypervisor crosscuts many subsystems.  This makes my work hyper-sensitive to breaks in Master.  The result is this nasty dilemma:  

Either argue against good recommendations, or lose a week getting a stable Master for QA cycle on the improvements.
 
Plus, as we've seen, when I fix Master, I introduce problems for people dependent on the bugs.

Not a great position to be in :(


DL

Re: Release Criteria

[Marcus Sorensen <sh...@gmail.com>]

I've repeated it several times in other threads, but I think at a bare
minimum it has to work in the listed versions of Ubuntu, CentOS (both
mgmt server and KVM host), XenServer, VmWare, what have you, along
with the advertised storage and networking. We can discuss limiting
that to OSS, maybe, as a languishing third party plugin shouldn't hold
us up, and if it's languishing and OSS we should perhaps consider
deprecation.  As seen in 4.2, we need to also make sure that upgrades
work according to a defined matrix (supported upgrade compatibility
has been discussed recently on the list as well, it has been a few
months). If we can test those things and pass before we cut an RC and
begin voting, I think we will be in far better shape than we have
been, from what I've seen. Maybe we don't stop there, but it seems
like a good place to start and basic enough to be doable. Maybe we
boil it down to a dozen or so checkboxes, being each supported
platform can successfully deploy a zone with one VM, each supported
storage type can deploy an instance, each of the supported upgrade
paths work. And beyond that, if there are bugs, they'll be addressed
in subsequent bugfix releases, but only things that affect those core
items are blockers.


On Wed, Sep 11, 2013 at 9:52 PM, Alex Huang <Al...@citrix.com> wrote:
>> >
>> > A broken master also slows down other devs. I can't remember the
>> number of times I've been debugging master for hours to find out something
>> broke it.
>> >
>>
>> so how do we enforce this ?
>>
>
> I'm so glad we raise this point.  For some time now, Prasanna, Amogh, Frank, and a number of others have been working on getting continuous integration working on CloudStack.  I've mentioned it before but because of the releases, the work's been delayed.  I think they are pretty close now so I like to propose it here.
>
> I've talked about it on here [1].  What do everyone think?  I specifically left out the bits about whether we should use gerrit.  That's more which system we use to implement the review part but I think we should get to the point where no matter who it is, their checkins are not committed to the master/release branch unless it has passed BVT/Regression tests.   That's the only way to ensure that master is always stable.
>
> --Alex
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Automated+Tests+Rules+and+Guidelines
>

RE: Release Criteria

[Alex Huang <Al...@citrix.com>]

> >
> > A broken master also slows down other devs. I can't remember the
> number of times I've been debugging master for hours to find out something
> broke it.
> >
> 
> so how do we enforce this ?
> 

I'm so glad we raise this point.  For some time now, Prasanna, Amogh, Frank, and a number of others have been working on getting continuous integration working on CloudStack.  I've mentioned it before but because of the releases, the work's been delayed.  I think they are pretty close now so I like to propose it here.

I've talked about it on here [1].  What do everyone think?  I specifically left out the bits about whether we should use gerrit.  That's more which system we use to implement the review part but I think we should get to the point where no matter who it is, their checkins are not committed to the master/release branch unless it has passed BVT/Regression tests.   That's the only way to ensure that master is always stable.

--Alex
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Automated+Tests+Rules+and+Guidelines

Re: Release Criteria

[Wido den Hollander <wi...@widodh.nl>]

On 09/11/2013 10:06 AM, Sebastien Goasguen wrote:
>
> On Sep 11, 2013, at 2:57 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>
>>
>>
>> On 09/11/2013 07:43 AM, Darren Shepherd wrote:
>>> On Tue, Sep 10, 2013 at 9:52 PM, Prasanna Santhanam <ts...@apache.org> wrote:
>>>
>>>>
>>>> I think we messed up with the users again this time. Partly a fault
>>>> that we can't get beta-quality builds for users to test. Seeing
>>>> everyone run 4.2 packages after release announcement and reporting
>>>> critical bugs I wish could've happened soon after freeze and during
>>>> the test schedule.
>>>>
>>>> To get beta quality builds we need to absolutely treat the master
>>>> branch as 'stable'. Never hurt it, automate against it, branch off
>>>> quality builds from it and create packages and mirror them. That'll
>>>> save us a ton of effort.
>>>>
>>>>
>>> Just to reinforce that branching point.  For all practical purposes, master
>>> should be treated like a release branch.  You only commit to master AFTER
>>> you've done QA.  Builds from master should be tested for only
>>> re-verification (or run your automated regression tests, since you
>>> obviously created those when you did QA) and for integration testing.
>>>
>>> So, I have no clue how people are doing QA, but if your checking into
>>> master when you think you're "dev done," so that QA can pickup a build and
>>> start testing, then your doing it all wrong.  Check into branch, test from
>>> branch, when all is swell, merge to master, revalidate on master.
>>>
>>
>> I think that's a good point. Master is not a playground. Stuff which goes in there should work.
>>
>> A broken master also slows down other devs. I can't remember the number of times I've been debugging master for hours to find out something broke it.
>>
>
> so how do we enforce this ?
>

Good question, but I think eduction is one of them.

It's just like having good commit message and having your Author en 
E-Mail set up properly in Git.

>> Wido
>>
>>> Darren
>>>
>

Re: Release Criteria

[Darren Shepherd <da...@gmail.com>]

Hmmm... well shoot.  If what I'm saying is an argument for gerrit, then I
take it back.  I don't like heavy processes.  Keep it simple, work with
good people.

Darren


On Wed, Sep 11, 2013 at 8:23 PM, Alex Huang <Al...@citrix.com> wrote:

> > Well in the apache model committers are nominated.  So basically we
> should
> > trust our committers.  So I'm going to say we enforce this by having good
> > discipline.  In really not a fan of adding more process.  In communities
> where
> > gerrit is used its usually done in a model where anybody can commit, so
> > you're forced to have a rigid process.  We should take nominating
> committers
> > seriously and committers should know best.
>
> Darren,
>
> I think you're actually malomg the argument for gerrit is the right
> process for us.  You're assuming ACS committership is granted based on code
>  but, in actuality, it is granted based on participation and a bit of code.
>  It's been discussed before.  Given that, then it just makes sense that all
> code contribution should be treated the same way with reviews.  Gerrit is a
> better way to implement this process than the review board.
>
> --Alex
>

RE: Release Criteria

[Alex Huang <Al...@citrix.com>]

> Well in the apache model committers are nominated.  So basically we should
> trust our committers.  So I'm going to say we enforce this by having good
> discipline.  In really not a fan of adding more process.  In communities where
> gerrit is used its usually done in a model where anybody can commit, so
> you're forced to have a rigid process.  We should take nominating committers
> seriously and committers should know best.

Darren,

I think you're actually malomg the argument for gerrit is the right process for us.  You're assuming ACS committership is granted based on code  but, in actuality, it is granted based on participation and a bit of code.  It's been discussed before.  Given that, then it just makes sense that all code contribution should be treated the same way with reviews.  Gerrit is a better way to implement this process than the review board.

--Alex

Re: Release Criteria

[Darren Shepherd <da...@gmail.com>]

Well in the apache model committers are nominated.  So basically we should trust our committers.  So I'm going to say we enforce this by having good discipline.  In really not a fan of adding more process.  In communities where gerrit is used its usually done in a model where anybody can commit, so you're forced to have a rigid process.  We should take nominating committers seriously and committers should know best.  

Frankly, for how young ACS is, I'm surprised at the number of committers.  I always hate when people gauge the strength of a community by the number of committers.  Cause everybody knows the way to get things done more efficiently is to add more opinions...

Darren

On Sep 11, 2013, at 1:06 AM, Sebastien Goasguen <ru...@gmail.com> wrote:

> 
> On Sep 11, 2013, at 2:57 AM, Wido den Hollander <wi...@widodh.nl> wrote:
> 
>> 
>> 
>> On 09/11/2013 07:43 AM, Darren Shepherd wrote:
>>> On Tue, Sep 10, 2013 at 9:52 PM, Prasanna Santhanam <ts...@apache.org> wrote:
>>> 
>>>> 
>>>> I think we messed up with the users again this time. Partly a fault
>>>> that we can't get beta-quality builds for users to test. Seeing
>>>> everyone run 4.2 packages after release announcement and reporting
>>>> critical bugs I wish could've happened soon after freeze and during
>>>> the test schedule.
>>>> 
>>>> To get beta quality builds we need to absolutely treat the master
>>>> branch as 'stable'. Never hurt it, automate against it, branch off
>>>> quality builds from it and create packages and mirror them. That'll
>>>> save us a ton of effort.
>>> Just to reinforce that branching point.  For all practical purposes, master
>>> should be treated like a release branch.  You only commit to master AFTER
>>> you've done QA.  Builds from master should be tested for only
>>> re-verification (or run your automated regression tests, since you
>>> obviously created those when you did QA) and for integration testing.
>>> 
>>> So, I have no clue how people are doing QA, but if your checking into
>>> master when you think you're "dev done," so that QA can pickup a build and
>>> start testing, then your doing it all wrong.  Check into branch, test from
>>> branch, when all is swell, merge to master, revalidate on master.
>> 
>> I think that's a good point. Master is not a playground. Stuff which goes in there should work.
>> 
>> A broken master also slows down other devs. I can't remember the number of times I've been debugging master for hours to find out something broke it.
> 
> so how do we enforce this ?
> 
>> Wido
>> 
>>> Darren
> 

Re: Release Criteria

[Dave Cahill <dc...@midokura.com>]

>
> > A broken master also slows down other devs. I can't remember the number
> of times I've been debugging master for hours to find out something broke
> it.
> >
> so how do we enforce this ?


IMO, Gerrit can be used to enforce a saner workflow, see previous
discussion at [1].

Having a workflow where you are explicitly encouraged to push directly to
the repo without review encourages half-baked commits.

>From the docs I've read, it seems this might be an Apache policy rather
than a CloudStack policy - Apache's new committer doc [2] seems to
encourage lack of review:
"Rather than creating a patch and submitting it to be actively reviewed and
then (hopefully) committed, you can now create a local patch and commit it
yourself"

In fairness, it does also say "Your patches will still be reviewed by your
fellow committers. This will happen after the event", but in the CloudStack
case, many people will have lost hours debugging your change by the time
(and if) that happens.

[1] http://markmail.org/message/inerurmjtc6v57ba
[2]
http://www.apache.org/dev/new-committers-guide.html#guide-for-new-committers


On Wed, Sep 11, 2013 at 5:06 PM, Sebastien Goasguen <ru...@gmail.com>wrote:

>
> On Sep 11, 2013, at 2:57 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>
> >
> >
> > On 09/11/2013 07:43 AM, Darren Shepherd wrote:
> >> On Tue, Sep 10, 2013 at 9:52 PM, Prasanna Santhanam <ts...@apache.org>
> wrote:
> >>
> >>>
> >>> I think we messed up with the users again this time. Partly a fault
> >>> that we can't get beta-quality builds for users to test. Seeing
> >>> everyone run 4.2 packages after release announcement and reporting
> >>> critical bugs I wish could've happened soon after freeze and during
> >>> the test schedule.
> >>>
> >>> To get beta quality builds we need to absolutely treat the master
> >>> branch as 'stable'. Never hurt it, automate against it, branch off
> >>> quality builds from it and create packages and mirror them. That'll
> >>> save us a ton of effort.
> >>>
> >>>
> >> Just to reinforce that branching point.  For all practical purposes,
> master
> >> should be treated like a release branch.  You only commit to master
> AFTER
> >> you've done QA.  Builds from master should be tested for only
> >> re-verification (or run your automated regression tests, since you
> >> obviously created those when you did QA) and for integration testing.
> >>
> >> So, I have no clue how people are doing QA, but if your checking into
> >> master when you think you're "dev done," so that QA can pickup a build
> and
> >> start testing, then your doing it all wrong.  Check into branch, test
> from
> >> branch, when all is swell, merge to master, revalidate on master.
> >>
> >
> > I think that's a good point. Master is not a playground. Stuff which
> goes in there should work.
> >
> > A broken master also slows down other devs. I can't remember the number
> of times I've been debugging master for hours to find out something broke
> it.
> >
>
> so how do we enforce this ?
>
> > Wido
> >
> >> Darren
> >>
>
>

Re: Release Criteria

[Sebastien Goasguen <ru...@gmail.com>]

On Sep 11, 2013, at 2:57 AM, Wido den Hollander <wi...@widodh.nl> wrote:

> 
> 
> On 09/11/2013 07:43 AM, Darren Shepherd wrote:
>> On Tue, Sep 10, 2013 at 9:52 PM, Prasanna Santhanam <ts...@apache.org> wrote:
>> 
>>> 
>>> I think we messed up with the users again this time. Partly a fault
>>> that we can't get beta-quality builds for users to test. Seeing
>>> everyone run 4.2 packages after release announcement and reporting
>>> critical bugs I wish could've happened soon after freeze and during
>>> the test schedule.
>>> 
>>> To get beta quality builds we need to absolutely treat the master
>>> branch as 'stable'. Never hurt it, automate against it, branch off
>>> quality builds from it and create packages and mirror them. That'll
>>> save us a ton of effort.
>>> 
>>> 
>> Just to reinforce that branching point.  For all practical purposes, master
>> should be treated like a release branch.  You only commit to master AFTER
>> you've done QA.  Builds from master should be tested for only
>> re-verification (or run your automated regression tests, since you
>> obviously created those when you did QA) and for integration testing.
>> 
>> So, I have no clue how people are doing QA, but if your checking into
>> master when you think you're "dev done," so that QA can pickup a build and
>> start testing, then your doing it all wrong.  Check into branch, test from
>> branch, when all is swell, merge to master, revalidate on master.
>> 
> 
> I think that's a good point. Master is not a playground. Stuff which goes in there should work.
> 
> A broken master also slows down other devs. I can't remember the number of times I've been debugging master for hours to find out something broke it.
> 

so how do we enforce this ?

> Wido
> 
>> Darren
>> 

Re: Release Criteria

[Wido den Hollander <wi...@widodh.nl>]

On 09/11/2013 07:43 AM, Darren Shepherd wrote:
> On Tue, Sep 10, 2013 at 9:52 PM, Prasanna Santhanam <ts...@apache.org> wrote:
>
>>
>> I think we messed up with the users again this time. Partly a fault
>> that we can't get beta-quality builds for users to test. Seeing
>> everyone run 4.2 packages after release announcement and reporting
>> critical bugs I wish could've happened soon after freeze and during
>> the test schedule.
>>
>> To get beta quality builds we need to absolutely treat the master
>> branch as 'stable'. Never hurt it, automate against it, branch off
>> quality builds from it and create packages and mirror them. That'll
>> save us a ton of effort.
>>
>>
> Just to reinforce that branching point.  For all practical purposes, master
> should be treated like a release branch.  You only commit to master AFTER
> you've done QA.  Builds from master should be tested for only
> re-verification (or run your automated regression tests, since you
> obviously created those when you did QA) and for integration testing.
>
> So, I have no clue how people are doing QA, but if your checking into
> master when you think you're "dev done," so that QA can pickup a build and
> start testing, then your doing it all wrong.  Check into branch, test from
> branch, when all is swell, merge to master, revalidate on master.
>

I think that's a good point. Master is not a playground. Stuff which 
goes in there should work.

A broken master also slows down other devs. I can't remember the number 
of times I've been debugging master for hours to find out something 
broke it.

Wido

> Darren
>

Re: Release Criteria

[Darren Shepherd <da...@gmail.com>]

On Tue, Sep 10, 2013 at 9:52 PM, Prasanna Santhanam <ts...@apache.org> wrote:

>
> I think we messed up with the users again this time. Partly a fault
> that we can't get beta-quality builds for users to test. Seeing
> everyone run 4.2 packages after release announcement and reporting
> critical bugs I wish could've happened soon after freeze and during
> the test schedule.
>
> To get beta quality builds we need to absolutely treat the master
> branch as 'stable'. Never hurt it, automate against it, branch off
> quality builds from it and create packages and mirror them. That'll
> save us a ton of effort.
>
>
Just to reinforce that branching point.  For all practical purposes, master
should be treated like a release branch.  You only commit to master AFTER
you've done QA.  Builds from master should be tested for only
re-verification (or run your automated regression tests, since you
obviously created those when you did QA) and for integration testing.

So, I have no clue how people are doing QA, but if your checking into
master when you think you're "dev done," so that QA can pickup a build and
start testing, then your doing it all wrong.  Check into branch, test from
branch, when all is swell, merge to master, revalidate on master.

Darren

Re: Release Criteria

[Prasanna Santhanam <ts...@apache.org>]

On Tue, Sep 10, 2013 at 08:46:43PM -0400, David Nalley wrote:
> Hi folks,
> 
> One of the things I've been pondering of late is a set of release
> criteria. E.g. here is what CloudStack MUST do to be considered for
> release.
> 
> So as background there is a somewhat complex social contract that I
> think we informally enter with our users. People expect us to release
> when we project - and we (IMO) damage the credibility of the project
> by suggesting a date and then missing, sometimes by a wide margin. In
> reading some of the documentation that other folks have around
> Time-based releases, I really like one of the analogies that Ubuntu
> uses [1] which is that of a play in a theatre. Things may not be
> perfect, people may not absolutely know all of their lines, but
> tickets have been sold, the audience is standing outside waiting to
> come in, and it's a pretty drastic event for the show not to go on.

Hollywood has trailers and if the trailer sucks, I don't go to the
movie :)

I think we messed up with the users again this time. Partly a fault
that we can't get beta-quality builds for users to test. Seeing
everyone run 4.2 packages after release announcement and reporting
critical bugs I wish could've happened soon after freeze and during
the test schedule.

To get beta quality builds we need to absolutely treat the master
branch as 'stable'. Never hurt it, automate against it, branch off
quality builds from it and create packages and mirror them. That'll
save us a ton of effort.

> 
> At the same time, bringing multiple RCs forward which get voted down
> doesn't necessarily inspire confidence. Some of this is caused by our
> lack of comprehensive testing. I think many people have an innate
> sense of what they think CloudStack should deliver in a product
> release; but we also have a number of people who suggest that they are
> timid in signing off on a product release simply because of the scope.
> I think we should actively be setting the expectations of consumers of
> our product. Particularly in the short term when we don't have
> completely comprehensive testing, having a standard that we can test
> to that is our 'minimum acceptable' makes sense in my mind.
> 
> I am going to spend a few days drafting a set of criteria, and I'll
> post it to the wiki, and ask for feedback and help in refining it,
> just wanted to give a heads up on what I am thinking, and hopefully
> get some consensus around it being a worthwhile thing.
> 
> --David
> 
> 
> 
> 
> 
> [1] https://wiki.ubuntu.com/TimeBasedReleases

-- 
Prasanna.,

------------------------
Powered by BigRock.com