[PROPOSAL] move away from time-based releases and/or revamp release process

[Marcus Sorensen <sh...@gmail.com>]

Guys,  I think we are not currently in a state to handle time-based
releases.  Until we can cut master at any time and have it releasable,
or at least at a reasonable RC-level matching minimum tested
requirements, it's just going to continue to be an exercise in
frustration to cut RCs simply because we hit a deadline.

Maybe we can get away with sticking to time-based if we revamp our
schedule and procedures, I don't know, but in light of how 4.1
(dragged on so long that some were seriously considering skipping/not
releasing it with 4.2 on its heels) and 4.2 (six rounds of votes so
far) have worked it's probably worth discussing.

Any suggestions on what might be better? It's been mentioned in the
past that it's a chicken-egg thing, many really don't try it until we
hit an RC, which causes multiple iterations. I do agree that many
don't take it seriously until we start cutting artifacts, but maybe we
do this in a more deliberate fashion instead of jumping right to the
vote. After feature/code freeze, cut some alpha artifacts, wait a
week, cut alpha2 or some beta artifacts, etc, and then at some point
anyone can propose that certain artifacts (or a new set of artifacts)
be put up for a vote as an RC. This gives us a way to signal that
we're gearing up for release and gives plenty of time for people to
test their components, or see the [PROPOSAL] and say 'oh crap, I had
better test my stuff', prior to cutting an RC.  Maybe this wouldn't
help in practice, but I think right now we go from telling the
community "code is frozen, don't check anything in unless its a bug
fix" to "here's our RC, try it out", without a formal testing window.
I realize the whole thing should be a testing window, but I don't
think it's conveyed well.

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[David Nalley <da...@gnsa.us>]

On Tue, Sep 24, 2013 at 12:06 PM, Chip Childers
<ch...@sungard.com> wrote:
> On Tue, Sep 24, 2013 at 10:00:32AM -0600, Marcus Sorensen wrote:
>> I guess I'm in the minority though, since we're the
>> only ones discussing it.
>
> No you are not.  Personally, I'm thinking about this problem a bit
> before adding my 2 cents.  Don't take that silence as thinking we are
> "ok".
>
> Short version of my thoughts:
>
> 1 - a schedule is really important to a project like this
> 2 - not releasing crap quality is really really important to a project
> like this
>
> Note the "really" vs. "really really".
>
> -chip

I agree with the sentiment in the above two points.
I, personally, do not think that the problem is with a time-based
release. I don't think that hurts or helps us in the quality
department.
I do think that we have to get to a point where we can unequivocally
verify the status of 'master' or a 'release' branch. I'd prefer that
we be able to do this before code gets pushed to it. The network
effect of even small changes in CloudStack can be profound. I am
almost to the point that I'd prefer us to focus the 4.3 release cycle
on nothing but building out QA infrastructure so that we can 'prove'
that proposed changes don't cause harm, and that the software is
actually releasable.

--David

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[Daan Hoogland <da...@gmail.com>]

Ilya,

I think you are in line with my 'on the side' notice. It is a promise I
like with one downside. You risk a successful 3rd party plugin writer still
and only supporting 5.6.5 whilst we are already at 7.3.x, hence the last
few lines of that remark.

So does not allowing a plugin 4.3 on a 4.2 instance also mean not allowing
a 4.2 on 4.3?
Can a developer build for 10.3 and up?
And how are we ensuring that at version 13 we disallow this plugin before
it is revisited by the author because we know of some quirk in our plugin
mechs that we created or fixed?
Rhetoric question, that last one. The answer being of course some extra
numbering scheme or table under the hood. It must be devised in my opinion
as allowing the plugin to load will hurt the user and the rep of ACS.
So answering the first two questions defines how much work this will be.

Daan


On Thu, Sep 26, 2013 at 8:15 PM, Musayev, Ilya <im...@webmd.net> wrote:

> Dan and David,
>
> I see your concerns. Let me explain what I mean by modular and how we can
> address dependency.
>
> Assume you have base ACS 4.2
> Then you would have plugins like Nicira 4.2.x, VMware 4.2.x, NetScaler
> 4.2.x etc.. Goes without saying, but you cannot use a plugin from 4.3.x
> family on ACS 4.2.x base...
> We can aslo introduce dependcy checks, such that if Nicira plugin is
> enabled, we need VMware plugin enabled as well.
>
> Why would we consider this? You can update modules more rapidly that
> include bug fixes (and perhaps minor non impacting enhacements) without
> waiting for the entire release cycle, you testing should be less as well,
> as you would only be testing affected modules, unless they strongly tie
> with another plugin.
>
> If properly architected, CloudStack wont suffer OpenStack's faith - which
> is what they go through now - of only specific version x.y.z module can
> work with another x.y.z module.
>
> Regards
> ilya
>
> > -----Original Message-----
> > From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> > Sent: Thursday, September 26, 2013 4:57 AM
> > To: dev
> > Subject: Re: [PROPOSAL] move away from time-based releases and/or
> > revamp release process
> >
> > Some more remarks on this:
> >
> > Another model is of keeping quality high is controlling what gets in per
> > release. This would go against the opensource model somewhat. You would
> > need e release manager like me and I strongly advice against that.
> >
> > Giving users more control on what goes into their system is a good
> thing. it
> > must not become a 'university degree required' hurdle though. It would
> > allow them to have a narrower view on what is good software,
> functionality -
> > and quality wise. That would lead to better quality of quality control.
> I know I
> > am no good at that aspect right now because of the load of code I need to
> > know about.
> >
> > <on the side>
> > hear hear, David. You are pointing at some of the challenges OSGi deals
> with.
> > Ours is a little simpler; As we don't need run-time control, our plugins
> can
> > have a strict hierarchical requirements tree which is somewhat easier to
> > handle. Still you might end up with a system with two hypervisor
> (versions)
> > that require a different storage or network version to work. or vice
> versa. I
> > think we should be happy at first if we get a static system where 3rd
> party
> > plugins work with version x till y. Note that then the plugin interface
> is to be
> > stable through a (semantically versioned) major release.
> >
> > challenge i like, :)
> > </straight on again>
> >
> > On Thu, Sep 26, 2013 at 6:21 AM, David Nalley <da...@gnsa.us> wrote:
> > > On Wed, Sep 25, 2013 at 11:26 AM, Musayev, Ilya
> > <im...@webmd.net> wrote:
> > >> We can still use scheduled release as we do now and yet have some
> > agility.
> > >>
> > >> An idea was passed around before if we can modularize ACS in future
> > releases VS being it monolithic.
> > >>
> > >> We can still have a core as is, but additional components/plugins can
> be
> > loaded adhoc pm the need base. As an example, there is no need to bake in
> > vmware support into release if a customer has not need for it. You cant
> just
> > upgrade vmware code unless you upgrade from version X.y.1 to X.y.2.
> > >>
> > >> Same applies to all other "plugins", that are not truly pluggable.
> > >>
> > >> Splitting components as separate "plugins" (or whatever the proper
> term
> > is) would ease the release cycle and give us flexibility in my opinion.
> > >>
> > >
> > > Can you imagine the complexity of that model? Core version 4.3.x has
> > > to work with VMware plugin 7.2.3, 7.1.5, and 7.0.2 and also has to
> > > work with Nicira plugin version 3.4.1, 3.5.2, and 3.6.0 - except that
> > > both of those plugins interact with each other as well as the core
> > > orchestration platform. We struggle to test well now, the additional
> > > combinations there boggle the mind.
> > >
> > > --David
>
>
>

RE: [PROPOSAL] move away from time-based releases and/or revamp release process

["Musayev, Ilya" <im...@webmd.net>]

Dan and David,

I see your concerns. Let me explain what I mean by modular and how we can address dependency.

Assume you have base ACS 4.2
Then you would have plugins like Nicira 4.2.x, VMware 4.2.x, NetScaler 4.2.x etc.. Goes without saying, but you cannot use a plugin from 4.3.x family on ACS 4.2.x base...
We can aslo introduce dependcy checks, such that if Nicira plugin is enabled, we need VMware plugin enabled as well.

Why would we consider this? You can update modules more rapidly that include bug fixes (and perhaps minor non impacting enhacements) without waiting for the entire release cycle, you testing should be less as well, as you would only be testing affected modules, unless they strongly tie with another plugin.

If properly architected, CloudStack wont suffer OpenStack's faith - which is what they go through now - of only specific version x.y.z module can work with another x.y.z module.

Regards
ilya

> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Thursday, September 26, 2013 4:57 AM
> To: dev
> Subject: Re: [PROPOSAL] move away from time-based releases and/or
> revamp release process
> 
> Some more remarks on this:
> 
> Another model is of keeping quality high is controlling what gets in per
> release. This would go against the opensource model somewhat. You would
> need e release manager like me and I strongly advice against that.
> 
> Giving users more control on what goes into their system is a good thing. it
> must not become a 'university degree required' hurdle though. It would
> allow them to have a narrower view on what is good software, functionality -
> and quality wise. That would lead to better quality of quality control. I know I
> am no good at that aspect right now because of the load of code I need to
> know about.
> 
> <on the side>
> hear hear, David. You are pointing at some of the challenges OSGi deals with.
> Ours is a little simpler; As we don't need run-time control, our plugins can
> have a strict hierarchical requirements tree which is somewhat easier to
> handle. Still you might end up with a system with two hypervisor (versions)
> that require a different storage or network version to work. or vice versa. I
> think we should be happy at first if we get a static system where 3rd party
> plugins work with version x till y. Note that then the plugin interface is to be
> stable through a (semantically versioned) major release.
> 
> challenge i like, :)
> </straight on again>
> 
> On Thu, Sep 26, 2013 at 6:21 AM, David Nalley <da...@gnsa.us> wrote:
> > On Wed, Sep 25, 2013 at 11:26 AM, Musayev, Ilya
> <im...@webmd.net> wrote:
> >> We can still use scheduled release as we do now and yet have some
> agility.
> >>
> >> An idea was passed around before if we can modularize ACS in future
> releases VS being it monolithic.
> >>
> >> We can still have a core as is, but additional components/plugins can be
> loaded adhoc pm the need base. As an example, there is no need to bake in
> vmware support into release if a customer has not need for it. You cant just
> upgrade vmware code unless you upgrade from version X.y.1 to X.y.2.
> >>
> >> Same applies to all other "plugins", that are not truly pluggable.
> >>
> >> Splitting components as separate "plugins" (or whatever the proper term
> is) would ease the release cycle and give us flexibility in my opinion.
> >>
> >
> > Can you imagine the complexity of that model? Core version 4.3.x has
> > to work with VMware plugin 7.2.3, 7.1.5, and 7.0.2 and also has to
> > work with Nicira plugin version 3.4.1, 3.5.2, and 3.6.0 - except that
> > both of those plugins interact with each other as well as the core
> > orchestration platform. We struggle to test well now, the additional
> > combinations there boggle the mind.
> >
> > --David

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[Daan Hoogland <da...@gmail.com>]

Some more remarks on this:

Another model is of keeping quality high is controlling what gets in
per release. This would go against the opensource model somewhat. You
would need e release manager like me and I strongly advice against
that.

Giving users more control on what goes into their system is a good
thing. it must not become a 'university degree required' hurdle
though. It would allow them to have a narrower view on what is good
software, functionality - and quality wise. That would lead to better
quality of quality control. I know I am no good at that aspect right
now because of the load of code I need to know about.

<on the side>
hear hear, David. You are pointing at some of the challenges OSGi
deals with. Ours is a little simpler; As we don't need run-time
control, our plugins can have a strict hierarchical requirements tree
which is somewhat easier to handle. Still you might end up with a
system with two hypervisor (versions) that require a different storage
or network version to work. or vice versa. I think we should be happy
at first if we get a static system where 3rd party plugins work with
version x till y. Note that then the plugin interface is to be stable
through a (semantically versioned) major release.

challenge i like, :)
</straight on again>

On Thu, Sep 26, 2013 at 6:21 AM, David Nalley <da...@gnsa.us> wrote:
> On Wed, Sep 25, 2013 at 11:26 AM, Musayev, Ilya <im...@webmd.net> wrote:
>> We can still use scheduled release as we do now and yet have some agility.
>>
>> An idea was passed around before if we can modularize ACS in future releases VS being it monolithic.
>>
>> We can still have a core as is, but additional components/plugins can be loaded adhoc pm the need base. As an example, there is no need to bake in vmware support into release if a customer has not need for it. You cant just upgrade vmware code unless you upgrade from version X.y.1 to X.y.2.
>>
>> Same applies to all other "plugins", that are not truly pluggable.
>>
>> Splitting components as separate "plugins" (or whatever the proper term is) would ease the release cycle and give us flexibility in my opinion.
>>
>
> Can you imagine the complexity of that model? Core version 4.3.x has
> to work with VMware plugin 7.2.3, 7.1.5, and 7.0.2 and also has to
> work with Nicira plugin version 3.4.1, 3.5.2, and 3.6.0 - except that
> both of those plugins interact with each other as well as the core
> orchestration platform. We struggle to test well now, the additional
> combinations there boggle the mind.
>
> --David

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[David Nalley <da...@gnsa.us>]

On Wed, Sep 25, 2013 at 11:26 AM, Musayev, Ilya <im...@webmd.net> wrote:
> We can still use scheduled release as we do now and yet have some agility.
>
> An idea was passed around before if we can modularize ACS in future releases VS being it monolithic.
>
> We can still have a core as is, but additional components/plugins can be loaded adhoc pm the need base. As an example, there is no need to bake in vmware support into release if a customer has not need for it. You cant just upgrade vmware code unless you upgrade from version X.y.1 to X.y.2.
>
> Same applies to all other "plugins", that are not truly pluggable.
>
> Splitting components as separate "plugins" (or whatever the proper term is) would ease the release cycle and give us flexibility in my opinion.
>

Can you imagine the complexity of that model? Core version 4.3.x has
to work with VMware plugin 7.2.3, 7.1.5, and 7.0.2 and also has to
work with Nicira plugin version 3.4.1, 3.5.2, and 3.6.0 - except that
both of those plugins interact with each other as well as the core
orchestration platform. We struggle to test well now, the additional
combinations there boggle the mind.

--David

RE: [PROPOSAL] move away from time-based releases and/or revamp release process

["Musayev, Ilya" <im...@webmd.net>]

We can still use scheduled release as we do now and yet have some agility.

An idea was passed around before if we can modularize ACS in future releases VS being it monolithic.

We can still have a core as is, but additional components/plugins can be loaded adhoc pm the need base. As an example, there is no need to bake in vmware support into release if a customer has not need for it. You cant just upgrade vmware code unless you upgrade from version X.y.1 to X.y.2. 

Same applies to all other "plugins", that are not truly pluggable.

Splitting components as separate "plugins" (or whatever the proper term is) would ease the release cycle and give us flexibility in my opinion.




> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Tuesday, September 24, 2013 12:22 PM
> To: dev
> Subject: Re: [PROPOSAL] move away from time-based releases and/or
> revamp release process
> 
> is this work a workshop to get at least a core group of us in line. We all have
> ideas and a lot of good ones are buried under an ton weighing archive of
> mails.
> 
> Daan
> 
> On Tue, Sep 24, 2013 at 6:06 PM, Chip Childers <ch...@sungard.com>
> wrote:
> > On Tue, Sep 24, 2013 at 10:00:32AM -0600, Marcus Sorensen wrote:
> >> I guess I'm in the minority though, since we're the only ones
> >> discussing it.
> >
> > No you are not.  Personally, I'm thinking about this problem a bit
> > before adding my 2 cents.  Don't take that silence as thinking we are
> > "ok".
> >
> > Short version of my thoughts:
> >
> > 1 - a schedule is really important to a project like this
> > 2 - not releasing crap quality is really really important to a project
> > like this
> >
> > Note the "really" vs. "really really".
> >
> > -chip

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[Daan Hoogland <da...@gmail.com>]

is this work a workshop to get at least a core group of us in line. We
all have ideas and a lot of good ones are buried under an ton weighing
archive of mails.

Daan

On Tue, Sep 24, 2013 at 6:06 PM, Chip Childers
<ch...@sungard.com> wrote:
> On Tue, Sep 24, 2013 at 10:00:32AM -0600, Marcus Sorensen wrote:
>> I guess I'm in the minority though, since we're the
>> only ones discussing it.
>
> No you are not.  Personally, I'm thinking about this problem a bit
> before adding my 2 cents.  Don't take that silence as thinking we are
> "ok".
>
> Short version of my thoughts:
>
> 1 - a schedule is really important to a project like this
> 2 - not releasing crap quality is really really important to a project
> like this
>
> Note the "really" vs. "really really".
>
> -chip

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[Chip Childers <ch...@sungard.com>]

On Tue, Sep 24, 2013 at 10:00:32AM -0600, Marcus Sorensen wrote:
> I guess I'm in the minority though, since we're the
> only ones discussing it.

No you are not.  Personally, I'm thinking about this problem a bit
before adding my 2 cents.  Don't take that silence as thinking we are
"ok".

Short version of my thoughts:

1 - a schedule is really important to a project like this
2 - not releasing crap quality is really really important to a project
like this

Note the "really" vs. "really really".

-chip

Fwd: [PROPOSAL] move away from time-based releases and/or revamp release process

[Mike Tutkowski <mi...@solidfire.com>]

Accidentally just sent this to Daan.

---------- Forwarded message ----------
From: Mike Tutkowski <mi...@solidfire.com>
Date: Wed, Sep 25, 2013 at 8:34 AM
Subject: Re: [PROPOSAL] move away from time-based releases and/or revamp
release process
To: Daan Hoogland <da...@gmail.com>


I think that would be a really good use of a Hackathon Day at CCC.


On Wed, Sep 25, 2013 at 5:27 AM, Daan Hoogland <da...@gmail.com>wrote:

> So you are willing to spend a hackathon on that in november in Amsterdam?
>
> @Prasanna: can we expect you with your invalued input on this subject
> there?
>
> I would really feel a lot of people in the community and in Citrix
> would sleep better if we have this rolling more smoothly.
>
> On Tue, Sep 24, 2013 at 11:20 PM, Mike Tutkowski
> <mi...@solidfire.com> wrote:
> > I think a distributed Jenkins setup would be great.
> >
> > If we had really awesome test coverage, I would be less frightened of
> > last-minute checkins, as well. :)
> >
> >
> > On Tue, Sep 24, 2013 at 3:17 PM, Daan Hoogland <daan.hoogland@gmail.com
> >wrote:
> >
> >> Mike, rest assured you and Marcus are not the only ones. More guarantee
> on
> >> a stable master is a general concern. Personally I don't feel we need
> more
> >> control on what is in the next release, if we make unit tests and
> automated
> >> integration tests a priority. That is kind of a claim I do have 'the'
> >> solution, though not well cooked ;) It's going to take a while (a
> colleague
> >> said four or five releases) before we have a good enough test set and a
> >> smoothly running continuous integration test engine. I think we at least
> >> need the distributed Jenkins setup where you can run your own
> integration
> >> tests to make sure your invested logic remains intact. This of course
> being
> >> only part of 'all the' answers.
> >>
> >> regards,
> >>
> >>
> >> On Tue, Sep 24, 2013 at 9:09 PM, Mike Tutkowski <
> >> mike.tutkowski@solidfire.com> wrote:
> >>
> >> > I was a bit hesitant to keep pushing this because there doesn't seem
> to
> >> be
> >> > a lot of support for it, but - as Marcus pointed out - I was quite
> >> alarmed
> >> > by the number and criticality of bugs checked in right before we cut
> our
> >> > first RC for 4.2. We simply were not ready.
> >> >
> >> > To me, it felt like something one might do before one gets out a
> decent
> >> > beta release.
> >> >
> >> > I certainly don't claim to have all the answers for this, but I do
> think
> >> we
> >> > need to develop some kind of a process whereby very few changes are
> made
> >> > immediately prior (like a month) to the first cut of a RC. We might
> even
> >> > need to discuss such changes as a community before they get checked in
> >> > (after a certain point).
> >> >
> >> > As far as master not always being usable, this is a serious problem,
> as
> >> > well.
> >> >
> >> > For example, I've been having trouble getting KVM to work and - in the
> >> > meanwhile - my code has fallen out of date with master over the past
> week
> >> > or so. However, I'm always afraid if I update from master while in the
> >> > middle of solving one problem that I'll have more problems to deal
> with
> >> > before I can get back to the initial problem (because something didn't
> >> work
> >> > in master).
> >> >
> >> > Again, I don't claim to have any solution for this problem, but I am
> >> happy
> >> > to help brainstorm.
> >> >
> >> >
> >> > On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen <
> shadowsor@gmail.com
> >> > >wrote:
> >> >
> >> > > On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi
> >> > > <an...@citrix.com> wrote:
> >> > > >
> >> > > >
> >> > > >> -----Original Message-----
> >> > > >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> >> > > >> Sent: Monday, September 23, 2013 12:25 PM
> >> > > >> To: dev@cloudstack.apache.org
> >> > > >> Subject: RE: [PROPOSAL] move away from time-based releases and/or
> >> > revamp
> >> > > >> release process
> >> > > >>
> >> > > >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi"
> >> > > >> <an...@citrix.com>
> >> > > >> wrote:
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > > -----Original Message-----
> >> > > >> > > From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> >> > > >> > > Sent: Monday, September 23, 2013 11:38 AM
> >> > > >> > > To: dev@cloudstack.apache.org
> >> > > >> > > Subject: [PROPOSAL] move away from time-based releases and/or
> >> > revamp
> >> > > >> > > release process
> >> > > >> > >
> >> > > >> > > Guys,  I think we are not currently in a state to handle
> >> > time-based
> >> > > >> > > releases.  Until we can cut master at any time and have it
> >> > > >> > > releasable, or at least at a reasonable RC-level matching
> >> minimum
> >> > > >> > > tested requirements, it's just going to continue to be an
> >> exercise
> >> > > >> > > in frustration to cut RCs simply because we hit a deadline.
> >> > > >> > [Animesh>] David is going to propose Release Criterion up for
> >> > > >> > discussion
> >> > > >> as per his thread [1]
> >> > > >>
> >> > > >> I see that thread more about defining what minimum bar we should
> >> > always
> >> > > >> have master at in order to meet time-based releases. Its where we
> >> want
> >> > > >> to go, but not what to do in the meantime.
> >> > > > [Animesh>] His proposal is not just for master, but also for
> deciding
> >> > > the release exit criterion and IMO is something we should follow for
> >> > 4.3.0
> >> > > and onwards
> >> > >
> >> > > Yes, I know. What I meant was that it will be a step toward
> >> > > stabilizing master, until we do that I'm not convinced we can adhere
> >> > > to any time-based expectation). It still doesn't fix our issue if
> >> > > we're going to insist on time-based releases, it just (from my
> >> > > undertanding) sets a bar for what is acceptable and what isn't, for
> >> > > any release. It stops the argument of "should we release with this
> >> > > bug".
> >> > >
> >> > > >>
> >> > > >> > >
> >> > > >> > > Maybe we can get away with sticking to time-based if we
> revamp
> >> our
> >> > > >> > > schedule and procedures, I don't know, but in light of how
> 4.1
> >> > > >> > > (dragged on so long that some were seriously considering
> >> > > >> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six
> >> > rounds
> >> > > >> > > of votes so
> >> > > >> > > far) have worked it's probably worth discussing.
> >> > > >> > >
> >> > > >> > > Any suggestions on what might be better? It's been mentioned
> in
> >> > the
> >> > > >> > > past that it's a chicken-egg thing, many really don't try it
> >> until
> >> > > >> > > we hit an RC, which causes multiple iterations. I do agree
> that
> >> > many
> >> > > >> > > don't take it seriously until we start cutting artifacts, but
> >> > maybe
> >> > > >> > > we do this in a more deliberate fashion instead of jumping
> right
> >> > to
> >> > > >> > > the vote. After feature/code freeze, cut some alpha
> artifacts,
> >> > wait
> >> > > >> > > a week, cut alpha2 or some beta artifacts, etc, and then at
> some
> >> > > >> > > point anyone can propose that certain artifacts (or a new
> set of
> >> > > >> > > artifacts) be put up for a vote as an RC. This gives us a
> way to
> >> > > >> > > signal that we're gearing up for release and gives plenty of
> >> time
> >> > > >> > > for people to test their components, or see the [PROPOSAL]
> and
> >> say
> >> > > >> > > 'oh crap, I had better test my stuff', prior to cutting an
> RC.
> >> > > >> > > Maybe this wouldn't help in practice, but I think right now
> we
> >> go
> >> > > >> > > from telling the community "code is frozen, don't check
> anything
> >> > in
> >> > > >> > > unless its a bug fix" to "here's our RC, try it out",
> without a
> >> > > >> formal testing window.
> >> > > >> > > I realize the whole thing should be a testing window, but I
> >> don't
> >> > > >> > > think it's conveyed well.
> >> > > >> >
> >> > > >> > [Animesh>] After the code freeze is all the stabilization and
> >> > > >> > integration
> >> > > >> testing phase and has been documented at [2].  No one should be
> >> > waiting
> >> > > >> until the RC to test their components for the first time. It
> should
> >> be
> >> > > >> happening after code freeze.
> >> > > >>
> >> > > >> >
> >> > > >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm
> >> > > >> > [2]
> >> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases
> >> > > >> >
> >> > > >>
> >> > > >> Got it. As mentioned I realize that the whole time there is
> supposed
> >> > to
> >> > > >> be testing, but its not really working that way in practice.
> People
> >> > are
> >> > > >> volunteers, they forget where things are, or they dont want to
> mess
> >> > with
> >> > > >> it unless there is an indication that its semi-stable, and then
> >> > suddenly
> >> > > >> an RC is thrown over the fence and we go through iterations of
> RC.
> >> By
> >> > > >> the time the RC comes through we should be done testing and just
> >> > verify
> >> > > >> that someone's last minute bug fix didn't cause a regression or
> >> > > >> something.
> >> > > > [Animesh>] RC is not thrown in it is discussed as part of the
> release
> >> > > schedule.  After the code-freeze date everyone is expected to
> complete
> >> > > their integration testing by RC date. In fact I had sent numerous
> >> > reminders
> >> > > prior to the first RC starting from 2 weeks before the proposed RC
> >> date.
> >> > >
> >> > > That's not the point. The code is changing at a rapid pace. Mike,
> for
> >> > > example, commented on tons of critical fixes going in right up until
> >> > > the RC is cut. Then we cut some artifacts and give people 72 hours
> to
> >> > > test and buy off.   What I'm advocating is to lengthen the process,
> >> > > and not tie it to a timeline until we have better testing that
> >> > > stabilizes our master. At that time, when people can trust master
> >> > > remotely, then maybe individuals will take the time to poke at it
> >> > > prior to RC. Maybe that's a horrible idea, but let's at least talk
> >> > > about doing something until we're stable... or do we think we can
> >> > > accomplish that in a timely fashion?
> >> > >
> >> > > I think there are a few subgroups in our team here.  1) people whose
> >> > > job it is to develop on cloudstack, but don't really use it, 2)
> people
> >> > > who use cloudstack daily, and only do development to bugfix and/or
> add
> >> > > a pet feature. There may be some overlap for some individuals. This
> >> > > process might work great for individuals whose job it is to focus on
> >> > > cloudstack every single day and are tightly integrated with the
> >> > > massive changes, but the rest of us who consume cloudstack don't
> >> > > always have time to look at the big picture and focus on the
> unstable
> >> > > branches. We use the releases and focus on making the stable ones
> >> > > better and/or fixing/adding our pet features, until the next stable
> >> > > one comes around. Until the development branches stabilize I don't
> >> > > believe it will work for the users, they won't get involved until
> the
> >> > > end.
> >> > >
> >> > > For me, personally, it's a waste of time to even look at a branch
> that
> >> > > probably won't work due to sweeping changes that tend to occur
> between
> >> > > releases.  Make your core changes, add spring, replace the storage
> >> > > subsystem, whatever it is, and then I'll go back and see what it
> broke
> >> > > after the bugs are worked out in all of that. That's how group #2
> >> > > thinks, in general. And right now the only indicator that we're to
> >> > > that point is when we start talking RC, at which point I have a 3
> day
> >> > > window that I hopefully catch and have time to play with it.
> >> > >
> >> > > >
> >> > > >
> >> > >
> >> > > My impression from your responses Animesh is that you feel
> everything
> >> > > is fine as-is.  I don't know how anyone could think that given what
> >> > > we've seen over the last two releases, especially you who had to cut
> >> > > six RCs. We're blowing past our "time based releases", and trying to
> >> > > push through buggy releases (for some reason). My intent was to sum
> up
> >> > > and focus on some of the comments I've seen over the past few weeks
> >> > > about low/sporadic RC participation, major changes going on at the
> >> > > last minute, etc. I guess I'm in the minority though, since we're
> the
> >> > > only ones discussing it.
> >> > >
> >> >
> >> >
> >> >
> >> > --
> >> > *Mike Tutkowski*
> >> > *Senior CloudStack Developer, SolidFire Inc.*
> >> > e: mike.tutkowski@solidfire.com
> >> > o: 303.746.7302
> >> > Advancing the way the world uses the
> >> > cloud<http://solidfire.com/solution/overview/?video=play>
> >> > *™*
> >> >
> >>
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the
> > cloud<http://solidfire.com/solution/overview/?video=play>
> > *™*
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[Daan Hoogland <da...@gmail.com>]

So you are willing to spend a hackathon on that in november in Amsterdam?

@Prasanna: can we expect you with your invalued input on this subject there?

I would really feel a lot of people in the community and in Citrix
would sleep better if we have this rolling more smoothly.

On Tue, Sep 24, 2013 at 11:20 PM, Mike Tutkowski
<mi...@solidfire.com> wrote:
> I think a distributed Jenkins setup would be great.
>
> If we had really awesome test coverage, I would be less frightened of
> last-minute checkins, as well. :)
>
>
> On Tue, Sep 24, 2013 at 3:17 PM, Daan Hoogland <da...@gmail.com>wrote:
>
>> Mike, rest assured you and Marcus are not the only ones. More guarantee on
>> a stable master is a general concern. Personally I don't feel we need more
>> control on what is in the next release, if we make unit tests and automated
>> integration tests a priority. That is kind of a claim I do have 'the'
>> solution, though not well cooked ;) It's going to take a while (a colleague
>> said four or five releases) before we have a good enough test set and a
>> smoothly running continuous integration test engine. I think we at least
>> need the distributed Jenkins setup where you can run your own integration
>> tests to make sure your invested logic remains intact. This of course being
>> only part of 'all the' answers.
>>
>> regards,
>>
>>
>> On Tue, Sep 24, 2013 at 9:09 PM, Mike Tutkowski <
>> mike.tutkowski@solidfire.com> wrote:
>>
>> > I was a bit hesitant to keep pushing this because there doesn't seem to
>> be
>> > a lot of support for it, but - as Marcus pointed out - I was quite
>> alarmed
>> > by the number and criticality of bugs checked in right before we cut our
>> > first RC for 4.2. We simply were not ready.
>> >
>> > To me, it felt like something one might do before one gets out a decent
>> > beta release.
>> >
>> > I certainly don't claim to have all the answers for this, but I do think
>> we
>> > need to develop some kind of a process whereby very few changes are made
>> > immediately prior (like a month) to the first cut of a RC. We might even
>> > need to discuss such changes as a community before they get checked in
>> > (after a certain point).
>> >
>> > As far as master not always being usable, this is a serious problem, as
>> > well.
>> >
>> > For example, I've been having trouble getting KVM to work and - in the
>> > meanwhile - my code has fallen out of date with master over the past week
>> > or so. However, I'm always afraid if I update from master while in the
>> > middle of solving one problem that I'll have more problems to deal with
>> > before I can get back to the initial problem (because something didn't
>> work
>> > in master).
>> >
>> > Again, I don't claim to have any solution for this problem, but I am
>> happy
>> > to help brainstorm.
>> >
>> >
>> > On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen <shadowsor@gmail.com
>> > >wrote:
>> >
>> > > On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi
>> > > <an...@citrix.com> wrote:
>> > > >
>> > > >
>> > > >> -----Original Message-----
>> > > >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> > > >> Sent: Monday, September 23, 2013 12:25 PM
>> > > >> To: dev@cloudstack.apache.org
>> > > >> Subject: RE: [PROPOSAL] move away from time-based releases and/or
>> > revamp
>> > > >> release process
>> > > >>
>> > > >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi"
>> > > >> <an...@citrix.com>
>> > > >> wrote:
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > > -----Original Message-----
>> > > >> > > From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> > > >> > > Sent: Monday, September 23, 2013 11:38 AM
>> > > >> > > To: dev@cloudstack.apache.org
>> > > >> > > Subject: [PROPOSAL] move away from time-based releases and/or
>> > revamp
>> > > >> > > release process
>> > > >> > >
>> > > >> > > Guys,  I think we are not currently in a state to handle
>> > time-based
>> > > >> > > releases.  Until we can cut master at any time and have it
>> > > >> > > releasable, or at least at a reasonable RC-level matching
>> minimum
>> > > >> > > tested requirements, it's just going to continue to be an
>> exercise
>> > > >> > > in frustration to cut RCs simply because we hit a deadline.
>> > > >> > [Animesh>] David is going to propose Release Criterion up for
>> > > >> > discussion
>> > > >> as per his thread [1]
>> > > >>
>> > > >> I see that thread more about defining what minimum bar we should
>> > always
>> > > >> have master at in order to meet time-based releases. Its where we
>> want
>> > > >> to go, but not what to do in the meantime.
>> > > > [Animesh>] His proposal is not just for master, but also for deciding
>> > > the release exit criterion and IMO is something we should follow for
>> > 4.3.0
>> > > and onwards
>> > >
>> > > Yes, I know. What I meant was that it will be a step toward
>> > > stabilizing master, until we do that I'm not convinced we can adhere
>> > > to any time-based expectation). It still doesn't fix our issue if
>> > > we're going to insist on time-based releases, it just (from my
>> > > undertanding) sets a bar for what is acceptable and what isn't, for
>> > > any release. It stops the argument of "should we release with this
>> > > bug".
>> > >
>> > > >>
>> > > >> > >
>> > > >> > > Maybe we can get away with sticking to time-based if we revamp
>> our
>> > > >> > > schedule and procedures, I don't know, but in light of how 4.1
>> > > >> > > (dragged on so long that some were seriously considering
>> > > >> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six
>> > rounds
>> > > >> > > of votes so
>> > > >> > > far) have worked it's probably worth discussing.
>> > > >> > >
>> > > >> > > Any suggestions on what might be better? It's been mentioned in
>> > the
>> > > >> > > past that it's a chicken-egg thing, many really don't try it
>> until
>> > > >> > > we hit an RC, which causes multiple iterations. I do agree that
>> > many
>> > > >> > > don't take it seriously until we start cutting artifacts, but
>> > maybe
>> > > >> > > we do this in a more deliberate fashion instead of jumping right
>> > to
>> > > >> > > the vote. After feature/code freeze, cut some alpha artifacts,
>> > wait
>> > > >> > > a week, cut alpha2 or some beta artifacts, etc, and then at some
>> > > >> > > point anyone can propose that certain artifacts (or a new set of
>> > > >> > > artifacts) be put up for a vote as an RC. This gives us a way to
>> > > >> > > signal that we're gearing up for release and gives plenty of
>> time
>> > > >> > > for people to test their components, or see the [PROPOSAL] and
>> say
>> > > >> > > 'oh crap, I had better test my stuff', prior to cutting an RC.
>> > > >> > > Maybe this wouldn't help in practice, but I think right now we
>> go
>> > > >> > > from telling the community "code is frozen, don't check anything
>> > in
>> > > >> > > unless its a bug fix" to "here's our RC, try it out", without a
>> > > >> formal testing window.
>> > > >> > > I realize the whole thing should be a testing window, but I
>> don't
>> > > >> > > think it's conveyed well.
>> > > >> >
>> > > >> > [Animesh>] After the code freeze is all the stabilization and
>> > > >> > integration
>> > > >> testing phase and has been documented at [2].  No one should be
>> > waiting
>> > > >> until the RC to test their components for the first time. It should
>> be
>> > > >> happening after code freeze.
>> > > >>
>> > > >> >
>> > > >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm
>> > > >> > [2]
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases
>> > > >> >
>> > > >>
>> > > >> Got it. As mentioned I realize that the whole time there is supposed
>> > to
>> > > >> be testing, but its not really working that way in practice. People
>> > are
>> > > >> volunteers, they forget where things are, or they dont want to mess
>> > with
>> > > >> it unless there is an indication that its semi-stable, and then
>> > suddenly
>> > > >> an RC is thrown over the fence and we go through iterations of RC.
>> By
>> > > >> the time the RC comes through we should be done testing and just
>> > verify
>> > > >> that someone's last minute bug fix didn't cause a regression or
>> > > >> something.
>> > > > [Animesh>] RC is not thrown in it is discussed as part of the release
>> > > schedule.  After the code-freeze date everyone is expected to complete
>> > > their integration testing by RC date. In fact I had sent numerous
>> > reminders
>> > > prior to the first RC starting from 2 weeks before the proposed RC
>> date.
>> > >
>> > > That's not the point. The code is changing at a rapid pace. Mike, for
>> > > example, commented on tons of critical fixes going in right up until
>> > > the RC is cut. Then we cut some artifacts and give people 72 hours to
>> > > test and buy off.   What I'm advocating is to lengthen the process,
>> > > and not tie it to a timeline until we have better testing that
>> > > stabilizes our master. At that time, when people can trust master
>> > > remotely, then maybe individuals will take the time to poke at it
>> > > prior to RC. Maybe that's a horrible idea, but let's at least talk
>> > > about doing something until we're stable... or do we think we can
>> > > accomplish that in a timely fashion?
>> > >
>> > > I think there are a few subgroups in our team here.  1) people whose
>> > > job it is to develop on cloudstack, but don't really use it, 2) people
>> > > who use cloudstack daily, and only do development to bugfix and/or add
>> > > a pet feature. There may be some overlap for some individuals. This
>> > > process might work great for individuals whose job it is to focus on
>> > > cloudstack every single day and are tightly integrated with the
>> > > massive changes, but the rest of us who consume cloudstack don't
>> > > always have time to look at the big picture and focus on the unstable
>> > > branches. We use the releases and focus on making the stable ones
>> > > better and/or fixing/adding our pet features, until the next stable
>> > > one comes around. Until the development branches stabilize I don't
>> > > believe it will work for the users, they won't get involved until the
>> > > end.
>> > >
>> > > For me, personally, it's a waste of time to even look at a branch that
>> > > probably won't work due to sweeping changes that tend to occur between
>> > > releases.  Make your core changes, add spring, replace the storage
>> > > subsystem, whatever it is, and then I'll go back and see what it broke
>> > > after the bugs are worked out in all of that. That's how group #2
>> > > thinks, in general. And right now the only indicator that we're to
>> > > that point is when we start talking RC, at which point I have a 3 day
>> > > window that I hopefully catch and have time to play with it.
>> > >
>> > > >
>> > > >
>> > >
>> > > My impression from your responses Animesh is that you feel everything
>> > > is fine as-is.  I don't know how anyone could think that given what
>> > > we've seen over the last two releases, especially you who had to cut
>> > > six RCs. We're blowing past our "time based releases", and trying to
>> > > push through buggy releases (for some reason). My intent was to sum up
>> > > and focus on some of the comments I've seen over the past few weeks
>> > > about low/sporadic RC participation, major changes going on at the
>> > > last minute, etc. I guess I'm in the minority though, since we're the
>> > > only ones discussing it.
>> > >
>> >
>> >
>> >
>> > --
>> > *Mike Tutkowski*
>> > *Senior CloudStack Developer, SolidFire Inc.*
>> > e: mike.tutkowski@solidfire.com
>> > o: 303.746.7302
>> > Advancing the way the world uses the
>> > cloud<http://solidfire.com/solution/overview/?video=play>
>> > *™*
>> >
>>
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[Mike Tutkowski <mi...@solidfire.com>]

I think a distributed Jenkins setup would be great.

If we had really awesome test coverage, I would be less frightened of
last-minute checkins, as well. :)


On Tue, Sep 24, 2013 at 3:17 PM, Daan Hoogland <da...@gmail.com>wrote:

> Mike, rest assured you and Marcus are not the only ones. More guarantee on
> a stable master is a general concern. Personally I don't feel we need more
> control on what is in the next release, if we make unit tests and automated
> integration tests a priority. That is kind of a claim I do have 'the'
> solution, though not well cooked ;) It's going to take a while (a colleague
> said four or five releases) before we have a good enough test set and a
> smoothly running continuous integration test engine. I think we at least
> need the distributed Jenkins setup where you can run your own integration
> tests to make sure your invested logic remains intact. This of course being
> only part of 'all the' answers.
>
> regards,
>
>
> On Tue, Sep 24, 2013 at 9:09 PM, Mike Tutkowski <
> mike.tutkowski@solidfire.com> wrote:
>
> > I was a bit hesitant to keep pushing this because there doesn't seem to
> be
> > a lot of support for it, but - as Marcus pointed out - I was quite
> alarmed
> > by the number and criticality of bugs checked in right before we cut our
> > first RC for 4.2. We simply were not ready.
> >
> > To me, it felt like something one might do before one gets out a decent
> > beta release.
> >
> > I certainly don't claim to have all the answers for this, but I do think
> we
> > need to develop some kind of a process whereby very few changes are made
> > immediately prior (like a month) to the first cut of a RC. We might even
> > need to discuss such changes as a community before they get checked in
> > (after a certain point).
> >
> > As far as master not always being usable, this is a serious problem, as
> > well.
> >
> > For example, I've been having trouble getting KVM to work and - in the
> > meanwhile - my code has fallen out of date with master over the past week
> > or so. However, I'm always afraid if I update from master while in the
> > middle of solving one problem that I'll have more problems to deal with
> > before I can get back to the initial problem (because something didn't
> work
> > in master).
> >
> > Again, I don't claim to have any solution for this problem, but I am
> happy
> > to help brainstorm.
> >
> >
> > On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen <shadowsor@gmail.com
> > >wrote:
> >
> > > On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi
> > > <an...@citrix.com> wrote:
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> > > >> Sent: Monday, September 23, 2013 12:25 PM
> > > >> To: dev@cloudstack.apache.org
> > > >> Subject: RE: [PROPOSAL] move away from time-based releases and/or
> > revamp
> > > >> release process
> > > >>
> > > >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi"
> > > >> <an...@citrix.com>
> > > >> wrote:
> > > >> >
> > > >> >
> > > >> >
> > > >> > > -----Original Message-----
> > > >> > > From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> > > >> > > Sent: Monday, September 23, 2013 11:38 AM
> > > >> > > To: dev@cloudstack.apache.org
> > > >> > > Subject: [PROPOSAL] move away from time-based releases and/or
> > revamp
> > > >> > > release process
> > > >> > >
> > > >> > > Guys,  I think we are not currently in a state to handle
> > time-based
> > > >> > > releases.  Until we can cut master at any time and have it
> > > >> > > releasable, or at least at a reasonable RC-level matching
> minimum
> > > >> > > tested requirements, it's just going to continue to be an
> exercise
> > > >> > > in frustration to cut RCs simply because we hit a deadline.
> > > >> > [Animesh>] David is going to propose Release Criterion up for
> > > >> > discussion
> > > >> as per his thread [1]
> > > >>
> > > >> I see that thread more about defining what minimum bar we should
> > always
> > > >> have master at in order to meet time-based releases. Its where we
> want
> > > >> to go, but not what to do in the meantime.
> > > > [Animesh>] His proposal is not just for master, but also for deciding
> > > the release exit criterion and IMO is something we should follow for
> > 4.3.0
> > > and onwards
> > >
> > > Yes, I know. What I meant was that it will be a step toward
> > > stabilizing master, until we do that I'm not convinced we can adhere
> > > to any time-based expectation). It still doesn't fix our issue if
> > > we're going to insist on time-based releases, it just (from my
> > > undertanding) sets a bar for what is acceptable and what isn't, for
> > > any release. It stops the argument of "should we release with this
> > > bug".
> > >
> > > >>
> > > >> > >
> > > >> > > Maybe we can get away with sticking to time-based if we revamp
> our
> > > >> > > schedule and procedures, I don't know, but in light of how 4.1
> > > >> > > (dragged on so long that some were seriously considering
> > > >> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six
> > rounds
> > > >> > > of votes so
> > > >> > > far) have worked it's probably worth discussing.
> > > >> > >
> > > >> > > Any suggestions on what might be better? It's been mentioned in
> > the
> > > >> > > past that it's a chicken-egg thing, many really don't try it
> until
> > > >> > > we hit an RC, which causes multiple iterations. I do agree that
> > many
> > > >> > > don't take it seriously until we start cutting artifacts, but
> > maybe
> > > >> > > we do this in a more deliberate fashion instead of jumping right
> > to
> > > >> > > the vote. After feature/code freeze, cut some alpha artifacts,
> > wait
> > > >> > > a week, cut alpha2 or some beta artifacts, etc, and then at some
> > > >> > > point anyone can propose that certain artifacts (or a new set of
> > > >> > > artifacts) be put up for a vote as an RC. This gives us a way to
> > > >> > > signal that we're gearing up for release and gives plenty of
> time
> > > >> > > for people to test their components, or see the [PROPOSAL] and
> say
> > > >> > > 'oh crap, I had better test my stuff', prior to cutting an RC.
> > > >> > > Maybe this wouldn't help in practice, but I think right now we
> go
> > > >> > > from telling the community "code is frozen, don't check anything
> > in
> > > >> > > unless its a bug fix" to "here's our RC, try it out", without a
> > > >> formal testing window.
> > > >> > > I realize the whole thing should be a testing window, but I
> don't
> > > >> > > think it's conveyed well.
> > > >> >
> > > >> > [Animesh>] After the code freeze is all the stabilization and
> > > >> > integration
> > > >> testing phase and has been documented at [2].  No one should be
> > waiting
> > > >> until the RC to test their components for the first time. It should
> be
> > > >> happening after code freeze.
> > > >>
> > > >> >
> > > >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm
> > > >> > [2]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases
> > > >> >
> > > >>
> > > >> Got it. As mentioned I realize that the whole time there is supposed
> > to
> > > >> be testing, but its not really working that way in practice. People
> > are
> > > >> volunteers, they forget where things are, or they dont want to mess
> > with
> > > >> it unless there is an indication that its semi-stable, and then
> > suddenly
> > > >> an RC is thrown over the fence and we go through iterations of RC.
> By
> > > >> the time the RC comes through we should be done testing and just
> > verify
> > > >> that someone's last minute bug fix didn't cause a regression or
> > > >> something.
> > > > [Animesh>] RC is not thrown in it is discussed as part of the release
> > > schedule.  After the code-freeze date everyone is expected to complete
> > > their integration testing by RC date. In fact I had sent numerous
> > reminders
> > > prior to the first RC starting from 2 weeks before the proposed RC
> date.
> > >
> > > That's not the point. The code is changing at a rapid pace. Mike, for
> > > example, commented on tons of critical fixes going in right up until
> > > the RC is cut. Then we cut some artifacts and give people 72 hours to
> > > test and buy off.   What I'm advocating is to lengthen the process,
> > > and not tie it to a timeline until we have better testing that
> > > stabilizes our master. At that time, when people can trust master
> > > remotely, then maybe individuals will take the time to poke at it
> > > prior to RC. Maybe that's a horrible idea, but let's at least talk
> > > about doing something until we're stable... or do we think we can
> > > accomplish that in a timely fashion?
> > >
> > > I think there are a few subgroups in our team here.  1) people whose
> > > job it is to develop on cloudstack, but don't really use it, 2) people
> > > who use cloudstack daily, and only do development to bugfix and/or add
> > > a pet feature. There may be some overlap for some individuals. This
> > > process might work great for individuals whose job it is to focus on
> > > cloudstack every single day and are tightly integrated with the
> > > massive changes, but the rest of us who consume cloudstack don't
> > > always have time to look at the big picture and focus on the unstable
> > > branches. We use the releases and focus on making the stable ones
> > > better and/or fixing/adding our pet features, until the next stable
> > > one comes around. Until the development branches stabilize I don't
> > > believe it will work for the users, they won't get involved until the
> > > end.
> > >
> > > For me, personally, it's a waste of time to even look at a branch that
> > > probably won't work due to sweeping changes that tend to occur between
> > > releases.  Make your core changes, add spring, replace the storage
> > > subsystem, whatever it is, and then I'll go back and see what it broke
> > > after the bugs are worked out in all of that. That's how group #2
> > > thinks, in general. And right now the only indicator that we're to
> > > that point is when we start talking RC, at which point I have a 3 day
> > > window that I hopefully catch and have time to play with it.
> > >
> > > >
> > > >
> > >
> > > My impression from your responses Animesh is that you feel everything
> > > is fine as-is.  I don't know how anyone could think that given what
> > > we've seen over the last two releases, especially you who had to cut
> > > six RCs. We're blowing past our "time based releases", and trying to
> > > push through buggy releases (for some reason). My intent was to sum up
> > > and focus on some of the comments I've seen over the past few weeks
> > > about low/sporadic RC participation, major changes going on at the
> > > last minute, etc. I guess I'm in the minority though, since we're the
> > > only ones discussing it.
> > >
> >
> >
> >
> > --
> > *Mike Tutkowski*
> > *Senior CloudStack Developer, SolidFire Inc.*
> > e: mike.tutkowski@solidfire.com
> > o: 303.746.7302
> > Advancing the way the world uses the
> > cloud<http://solidfire.com/solution/overview/?video=play>
> > *™*
> >
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[Daan Hoogland <da...@gmail.com>]

Mike, rest assured you and Marcus are not the only ones. More guarantee on
a stable master is a general concern. Personally I don't feel we need more
control on what is in the next release, if we make unit tests and automated
integration tests a priority. That is kind of a claim I do have 'the'
solution, though not well cooked ;) It's going to take a while (a colleague
said four or five releases) before we have a good enough test set and a
smoothly running continuous integration test engine. I think we at least
need the distributed Jenkins setup where you can run your own integration
tests to make sure your invested logic remains intact. This of course being
only part of 'all the' answers.

regards,


On Tue, Sep 24, 2013 at 9:09 PM, Mike Tutkowski <
mike.tutkowski@solidfire.com> wrote:

> I was a bit hesitant to keep pushing this because there doesn't seem to be
> a lot of support for it, but - as Marcus pointed out - I was quite alarmed
> by the number and criticality of bugs checked in right before we cut our
> first RC for 4.2. We simply were not ready.
>
> To me, it felt like something one might do before one gets out a decent
> beta release.
>
> I certainly don't claim to have all the answers for this, but I do think we
> need to develop some kind of a process whereby very few changes are made
> immediately prior (like a month) to the first cut of a RC. We might even
> need to discuss such changes as a community before they get checked in
> (after a certain point).
>
> As far as master not always being usable, this is a serious problem, as
> well.
>
> For example, I've been having trouble getting KVM to work and - in the
> meanwhile - my code has fallen out of date with master over the past week
> or so. However, I'm always afraid if I update from master while in the
> middle of solving one problem that I'll have more problems to deal with
> before I can get back to the initial problem (because something didn't work
> in master).
>
> Again, I don't claim to have any solution for this problem, but I am happy
> to help brainstorm.
>
>
> On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen <shadowsor@gmail.com
> >wrote:
>
> > On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi
> > <an...@citrix.com> wrote:
> > >
> > >
> > >> -----Original Message-----
> > >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> > >> Sent: Monday, September 23, 2013 12:25 PM
> > >> To: dev@cloudstack.apache.org
> > >> Subject: RE: [PROPOSAL] move away from time-based releases and/or
> revamp
> > >> release process
> > >>
> > >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi"
> > >> <an...@citrix.com>
> > >> wrote:
> > >> >
> > >> >
> > >> >
> > >> > > -----Original Message-----
> > >> > > From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> > >> > > Sent: Monday, September 23, 2013 11:38 AM
> > >> > > To: dev@cloudstack.apache.org
> > >> > > Subject: [PROPOSAL] move away from time-based releases and/or
> revamp
> > >> > > release process
> > >> > >
> > >> > > Guys,  I think we are not currently in a state to handle
> time-based
> > >> > > releases.  Until we can cut master at any time and have it
> > >> > > releasable, or at least at a reasonable RC-level matching minimum
> > >> > > tested requirements, it's just going to continue to be an exercise
> > >> > > in frustration to cut RCs simply because we hit a deadline.
> > >> > [Animesh>] David is going to propose Release Criterion up for
> > >> > discussion
> > >> as per his thread [1]
> > >>
> > >> I see that thread more about defining what minimum bar we should
> always
> > >> have master at in order to meet time-based releases. Its where we want
> > >> to go, but not what to do in the meantime.
> > > [Animesh>] His proposal is not just for master, but also for deciding
> > the release exit criterion and IMO is something we should follow for
> 4.3.0
> > and onwards
> >
> > Yes, I know. What I meant was that it will be a step toward
> > stabilizing master, until we do that I'm not convinced we can adhere
> > to any time-based expectation). It still doesn't fix our issue if
> > we're going to insist on time-based releases, it just (from my
> > undertanding) sets a bar for what is acceptable and what isn't, for
> > any release. It stops the argument of "should we release with this
> > bug".
> >
> > >>
> > >> > >
> > >> > > Maybe we can get away with sticking to time-based if we revamp our
> > >> > > schedule and procedures, I don't know, but in light of how 4.1
> > >> > > (dragged on so long that some were seriously considering
> > >> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six
> rounds
> > >> > > of votes so
> > >> > > far) have worked it's probably worth discussing.
> > >> > >
> > >> > > Any suggestions on what might be better? It's been mentioned in
> the
> > >> > > past that it's a chicken-egg thing, many really don't try it until
> > >> > > we hit an RC, which causes multiple iterations. I do agree that
> many
> > >> > > don't take it seriously until we start cutting artifacts, but
> maybe
> > >> > > we do this in a more deliberate fashion instead of jumping right
> to
> > >> > > the vote. After feature/code freeze, cut some alpha artifacts,
> wait
> > >> > > a week, cut alpha2 or some beta artifacts, etc, and then at some
> > >> > > point anyone can propose that certain artifacts (or a new set of
> > >> > > artifacts) be put up for a vote as an RC. This gives us a way to
> > >> > > signal that we're gearing up for release and gives plenty of time
> > >> > > for people to test their components, or see the [PROPOSAL] and say
> > >> > > 'oh crap, I had better test my stuff', prior to cutting an RC.
> > >> > > Maybe this wouldn't help in practice, but I think right now we go
> > >> > > from telling the community "code is frozen, don't check anything
> in
> > >> > > unless its a bug fix" to "here's our RC, try it out", without a
> > >> formal testing window.
> > >> > > I realize the whole thing should be a testing window, but I don't
> > >> > > think it's conveyed well.
> > >> >
> > >> > [Animesh>] After the code freeze is all the stabilization and
> > >> > integration
> > >> testing phase and has been documented at [2].  No one should be
> waiting
> > >> until the RC to test their components for the first time. It should be
> > >> happening after code freeze.
> > >>
> > >> >
> > >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm
> > >> > [2] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases
> > >> >
> > >>
> > >> Got it. As mentioned I realize that the whole time there is supposed
> to
> > >> be testing, but its not really working that way in practice. People
> are
> > >> volunteers, they forget where things are, or they dont want to mess
> with
> > >> it unless there is an indication that its semi-stable, and then
> suddenly
> > >> an RC is thrown over the fence and we go through iterations of RC. By
> > >> the time the RC comes through we should be done testing and just
> verify
> > >> that someone's last minute bug fix didn't cause a regression or
> > >> something.
> > > [Animesh>] RC is not thrown in it is discussed as part of the release
> > schedule.  After the code-freeze date everyone is expected to complete
> > their integration testing by RC date. In fact I had sent numerous
> reminders
> > prior to the first RC starting from 2 weeks before the proposed RC date.
> >
> > That's not the point. The code is changing at a rapid pace. Mike, for
> > example, commented on tons of critical fixes going in right up until
> > the RC is cut. Then we cut some artifacts and give people 72 hours to
> > test and buy off.   What I'm advocating is to lengthen the process,
> > and not tie it to a timeline until we have better testing that
> > stabilizes our master. At that time, when people can trust master
> > remotely, then maybe individuals will take the time to poke at it
> > prior to RC. Maybe that's a horrible idea, but let's at least talk
> > about doing something until we're stable... or do we think we can
> > accomplish that in a timely fashion?
> >
> > I think there are a few subgroups in our team here.  1) people whose
> > job it is to develop on cloudstack, but don't really use it, 2) people
> > who use cloudstack daily, and only do development to bugfix and/or add
> > a pet feature. There may be some overlap for some individuals. This
> > process might work great for individuals whose job it is to focus on
> > cloudstack every single day and are tightly integrated with the
> > massive changes, but the rest of us who consume cloudstack don't
> > always have time to look at the big picture and focus on the unstable
> > branches. We use the releases and focus on making the stable ones
> > better and/or fixing/adding our pet features, until the next stable
> > one comes around. Until the development branches stabilize I don't
> > believe it will work for the users, they won't get involved until the
> > end.
> >
> > For me, personally, it's a waste of time to even look at a branch that
> > probably won't work due to sweeping changes that tend to occur between
> > releases.  Make your core changes, add spring, replace the storage
> > subsystem, whatever it is, and then I'll go back and see what it broke
> > after the bugs are worked out in all of that. That's how group #2
> > thinks, in general. And right now the only indicator that we're to
> > that point is when we start talking RC, at which point I have a 3 day
> > window that I hopefully catch and have time to play with it.
> >
> > >
> > >
> >
> > My impression from your responses Animesh is that you feel everything
> > is fine as-is.  I don't know how anyone could think that given what
> > we've seen over the last two releases, especially you who had to cut
> > six RCs. We're blowing past our "time based releases", and trying to
> > push through buggy releases (for some reason). My intent was to sum up
> > and focus on some of the comments I've seen over the past few weeks
> > about low/sporadic RC participation, major changes going on at the
> > last minute, etc. I guess I'm in the minority though, since we're the
> > only ones discussing it.
> >
>
>
>
> --
> *Mike Tutkowski*
> *Senior CloudStack Developer, SolidFire Inc.*
> e: mike.tutkowski@solidfire.com
> o: 303.746.7302
> Advancing the way the world uses the
> cloud<http://solidfire.com/solution/overview/?video=play>
> *™*
>

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[Mike Tutkowski <mi...@solidfire.com>]

I was a bit hesitant to keep pushing this because there doesn't seem to be
a lot of support for it, but - as Marcus pointed out - I was quite alarmed
by the number and criticality of bugs checked in right before we cut our
first RC for 4.2. We simply were not ready.

To me, it felt like something one might do before one gets out a decent
beta release.

I certainly don't claim to have all the answers for this, but I do think we
need to develop some kind of a process whereby very few changes are made
immediately prior (like a month) to the first cut of a RC. We might even
need to discuss such changes as a community before they get checked in
(after a certain point).

As far as master not always being usable, this is a serious problem, as
well.

For example, I've been having trouble getting KVM to work and - in the
meanwhile - my code has fallen out of date with master over the past week
or so. However, I'm always afraid if I update from master while in the
middle of solving one problem that I'll have more problems to deal with
before I can get back to the initial problem (because something didn't work
in master).

Again, I don't claim to have any solution for this problem, but I am happy
to help brainstorm.


On Tue, Sep 24, 2013 at 10:00 AM, Marcus Sorensen <sh...@gmail.com>wrote:

> On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi
> <an...@citrix.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> >> Sent: Monday, September 23, 2013 12:25 PM
> >> To: dev@cloudstack.apache.org
> >> Subject: RE: [PROPOSAL] move away from time-based releases and/or revamp
> >> release process
> >>
> >> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi"
> >> <an...@citrix.com>
> >> wrote:
> >> >
> >> >
> >> >
> >> > > -----Original Message-----
> >> > > From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> >> > > Sent: Monday, September 23, 2013 11:38 AM
> >> > > To: dev@cloudstack.apache.org
> >> > > Subject: [PROPOSAL] move away from time-based releases and/or revamp
> >> > > release process
> >> > >
> >> > > Guys,  I think we are not currently in a state to handle time-based
> >> > > releases.  Until we can cut master at any time and have it
> >> > > releasable, or at least at a reasonable RC-level matching minimum
> >> > > tested requirements, it's just going to continue to be an exercise
> >> > > in frustration to cut RCs simply because we hit a deadline.
> >> > [Animesh>] David is going to propose Release Criterion up for
> >> > discussion
> >> as per his thread [1]
> >>
> >> I see that thread more about defining what minimum bar we should always
> >> have master at in order to meet time-based releases. Its where we want
> >> to go, but not what to do in the meantime.
> > [Animesh>] His proposal is not just for master, but also for deciding
> the release exit criterion and IMO is something we should follow for 4.3.0
> and onwards
>
> Yes, I know. What I meant was that it will be a step toward
> stabilizing master, until we do that I'm not convinced we can adhere
> to any time-based expectation). It still doesn't fix our issue if
> we're going to insist on time-based releases, it just (from my
> undertanding) sets a bar for what is acceptable and what isn't, for
> any release. It stops the argument of "should we release with this
> bug".
>
> >>
> >> > >
> >> > > Maybe we can get away with sticking to time-based if we revamp our
> >> > > schedule and procedures, I don't know, but in light of how 4.1
> >> > > (dragged on so long that some were seriously considering
> >> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six rounds
> >> > > of votes so
> >> > > far) have worked it's probably worth discussing.
> >> > >
> >> > > Any suggestions on what might be better? It's been mentioned in the
> >> > > past that it's a chicken-egg thing, many really don't try it until
> >> > > we hit an RC, which causes multiple iterations. I do agree that many
> >> > > don't take it seriously until we start cutting artifacts, but maybe
> >> > > we do this in a more deliberate fashion instead of jumping right to
> >> > > the vote. After feature/code freeze, cut some alpha artifacts, wait
> >> > > a week, cut alpha2 or some beta artifacts, etc, and then at some
> >> > > point anyone can propose that certain artifacts (or a new set of
> >> > > artifacts) be put up for a vote as an RC. This gives us a way to
> >> > > signal that we're gearing up for release and gives plenty of time
> >> > > for people to test their components, or see the [PROPOSAL] and say
> >> > > 'oh crap, I had better test my stuff', prior to cutting an RC.
> >> > > Maybe this wouldn't help in practice, but I think right now we go
> >> > > from telling the community "code is frozen, don't check anything in
> >> > > unless its a bug fix" to "here's our RC, try it out", without a
> >> formal testing window.
> >> > > I realize the whole thing should be a testing window, but I don't
> >> > > think it's conveyed well.
> >> >
> >> > [Animesh>] After the code freeze is all the stabilization and
> >> > integration
> >> testing phase and has been documented at [2].  No one should be waiting
> >> until the RC to test their components for the first time. It should be
> >> happening after code freeze.
> >>
> >> >
> >> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm
> >> > [2] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases
> >> >
> >>
> >> Got it. As mentioned I realize that the whole time there is supposed to
> >> be testing, but its not really working that way in practice. People are
> >> volunteers, they forget where things are, or they dont want to mess with
> >> it unless there is an indication that its semi-stable, and then suddenly
> >> an RC is thrown over the fence and we go through iterations of RC. By
> >> the time the RC comes through we should be done testing and just verify
> >> that someone's last minute bug fix didn't cause a regression or
> >> something.
> > [Animesh>] RC is not thrown in it is discussed as part of the release
> schedule.  After the code-freeze date everyone is expected to complete
> their integration testing by RC date. In fact I had sent numerous reminders
> prior to the first RC starting from 2 weeks before the proposed RC date.
>
> That's not the point. The code is changing at a rapid pace. Mike, for
> example, commented on tons of critical fixes going in right up until
> the RC is cut. Then we cut some artifacts and give people 72 hours to
> test and buy off.   What I'm advocating is to lengthen the process,
> and not tie it to a timeline until we have better testing that
> stabilizes our master. At that time, when people can trust master
> remotely, then maybe individuals will take the time to poke at it
> prior to RC. Maybe that's a horrible idea, but let's at least talk
> about doing something until we're stable... or do we think we can
> accomplish that in a timely fashion?
>
> I think there are a few subgroups in our team here.  1) people whose
> job it is to develop on cloudstack, but don't really use it, 2) people
> who use cloudstack daily, and only do development to bugfix and/or add
> a pet feature. There may be some overlap for some individuals. This
> process might work great for individuals whose job it is to focus on
> cloudstack every single day and are tightly integrated with the
> massive changes, but the rest of us who consume cloudstack don't
> always have time to look at the big picture and focus on the unstable
> branches. We use the releases and focus on making the stable ones
> better and/or fixing/adding our pet features, until the next stable
> one comes around. Until the development branches stabilize I don't
> believe it will work for the users, they won't get involved until the
> end.
>
> For me, personally, it's a waste of time to even look at a branch that
> probably won't work due to sweeping changes that tend to occur between
> releases.  Make your core changes, add spring, replace the storage
> subsystem, whatever it is, and then I'll go back and see what it broke
> after the bugs are worked out in all of that. That's how group #2
> thinks, in general. And right now the only indicator that we're to
> that point is when we start talking RC, at which point I have a 3 day
> window that I hopefully catch and have time to play with it.
>
> >
> >
>
> My impression from your responses Animesh is that you feel everything
> is fine as-is.  I don't know how anyone could think that given what
> we've seen over the last two releases, especially you who had to cut
> six RCs. We're blowing past our "time based releases", and trying to
> push through buggy releases (for some reason). My intent was to sum up
> and focus on some of the comments I've seen over the past few weeks
> about low/sporadic RC participation, major changes going on at the
> last minute, etc. I guess I'm in the minority though, since we're the
> only ones discussing it.
>



-- 
*Mike Tutkowski*
*Senior CloudStack Developer, SolidFire Inc.*
e: mike.tutkowski@solidfire.com
o: 303.746.7302
Advancing the way the world uses the
cloud<http://solidfire.com/solution/overview/?video=play>
*™*

Re: [PROPOSAL] move away from time-based releases and/or revamp release process

[Marcus Sorensen <sh...@gmail.com>]

On Mon, Sep 23, 2013 at 1:55 PM, Animesh Chaturvedi
<an...@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> Sent: Monday, September 23, 2013 12:25 PM
>> To: dev@cloudstack.apache.org
>> Subject: RE: [PROPOSAL] move away from time-based releases and/or revamp
>> release process
>>
>> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi"
>> <an...@citrix.com>
>> wrote:
>> >
>> >
>> >
>> > > -----Original Message-----
>> > > From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> > > Sent: Monday, September 23, 2013 11:38 AM
>> > > To: dev@cloudstack.apache.org
>> > > Subject: [PROPOSAL] move away from time-based releases and/or revamp
>> > > release process
>> > >
>> > > Guys,  I think we are not currently in a state to handle time-based
>> > > releases.  Until we can cut master at any time and have it
>> > > releasable, or at least at a reasonable RC-level matching minimum
>> > > tested requirements, it's just going to continue to be an exercise
>> > > in frustration to cut RCs simply because we hit a deadline.
>> > [Animesh>] David is going to propose Release Criterion up for
>> > discussion
>> as per his thread [1]
>>
>> I see that thread more about defining what minimum bar we should always
>> have master at in order to meet time-based releases. Its where we want
>> to go, but not what to do in the meantime.
> [Animesh>] His proposal is not just for master, but also for deciding the release exit criterion and IMO is something we should follow for 4.3.0 and onwards

Yes, I know. What I meant was that it will be a step toward
stabilizing master, until we do that I'm not convinced we can adhere
to any time-based expectation). It still doesn't fix our issue if
we're going to insist on time-based releases, it just (from my
undertanding) sets a bar for what is acceptable and what isn't, for
any release. It stops the argument of "should we release with this
bug".

>>
>> > >
>> > > Maybe we can get away with sticking to time-based if we revamp our
>> > > schedule and procedures, I don't know, but in light of how 4.1
>> > > (dragged on so long that some were seriously considering
>> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six rounds
>> > > of votes so
>> > > far) have worked it's probably worth discussing.
>> > >
>> > > Any suggestions on what might be better? It's been mentioned in the
>> > > past that it's a chicken-egg thing, many really don't try it until
>> > > we hit an RC, which causes multiple iterations. I do agree that many
>> > > don't take it seriously until we start cutting artifacts, but maybe
>> > > we do this in a more deliberate fashion instead of jumping right to
>> > > the vote. After feature/code freeze, cut some alpha artifacts, wait
>> > > a week, cut alpha2 or some beta artifacts, etc, and then at some
>> > > point anyone can propose that certain artifacts (or a new set of
>> > > artifacts) be put up for a vote as an RC. This gives us a way to
>> > > signal that we're gearing up for release and gives plenty of time
>> > > for people to test their components, or see the [PROPOSAL] and say
>> > > 'oh crap, I had better test my stuff', prior to cutting an RC.
>> > > Maybe this wouldn't help in practice, but I think right now we go
>> > > from telling the community "code is frozen, don't check anything in
>> > > unless its a bug fix" to "here's our RC, try it out", without a
>> formal testing window.
>> > > I realize the whole thing should be a testing window, but I don't
>> > > think it's conveyed well.
>> >
>> > [Animesh>] After the code freeze is all the stabilization and
>> > integration
>> testing phase and has been documented at [2].  No one should be waiting
>> until the RC to test their components for the first time. It should be
>> happening after code freeze.
>>
>> >
>> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm
>> > [2] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases
>> >
>>
>> Got it. As mentioned I realize that the whole time there is supposed to
>> be testing, but its not really working that way in practice. People are
>> volunteers, they forget where things are, or they dont want to mess with
>> it unless there is an indication that its semi-stable, and then suddenly
>> an RC is thrown over the fence and we go through iterations of RC. By
>> the time the RC comes through we should be done testing and just verify
>> that someone's last minute bug fix didn't cause a regression or
>> something.
> [Animesh>] RC is not thrown in it is discussed as part of the release schedule.  After the code-freeze date everyone is expected to complete their integration testing by RC date. In fact I had sent numerous reminders prior to the first RC starting from 2 weeks before the proposed RC date.

That's not the point. The code is changing at a rapid pace. Mike, for
example, commented on tons of critical fixes going in right up until
the RC is cut. Then we cut some artifacts and give people 72 hours to
test and buy off.   What I'm advocating is to lengthen the process,
and not tie it to a timeline until we have better testing that
stabilizes our master. At that time, when people can trust master
remotely, then maybe individuals will take the time to poke at it
prior to RC. Maybe that's a horrible idea, but let's at least talk
about doing something until we're stable... or do we think we can
accomplish that in a timely fashion?

I think there are a few subgroups in our team here.  1) people whose
job it is to develop on cloudstack, but don't really use it, 2) people
who use cloudstack daily, and only do development to bugfix and/or add
a pet feature. There may be some overlap for some individuals. This
process might work great for individuals whose job it is to focus on
cloudstack every single day and are tightly integrated with the
massive changes, but the rest of us who consume cloudstack don't
always have time to look at the big picture and focus on the unstable
branches. We use the releases and focus on making the stable ones
better and/or fixing/adding our pet features, until the next stable
one comes around. Until the development branches stabilize I don't
believe it will work for the users, they won't get involved until the
end.

For me, personally, it's a waste of time to even look at a branch that
probably won't work due to sweeping changes that tend to occur between
releases.  Make your core changes, add spring, replace the storage
subsystem, whatever it is, and then I'll go back and see what it broke
after the bugs are worked out in all of that. That's how group #2
thinks, in general. And right now the only indicator that we're to
that point is when we start talking RC, at which point I have a 3 day
window that I hopefully catch and have time to play with it.

>
>

My impression from your responses Animesh is that you feel everything
is fine as-is.  I don't know how anyone could think that given what
we've seen over the last two releases, especially you who had to cut
six RCs. We're blowing past our "time based releases", and trying to
push through buggy releases (for some reason). My intent was to sum up
and focus on some of the comments I've seen over the past few weeks
about low/sporadic RC participation, major changes going on at the
last minute, etc. I guess I'm in the minority though, since we're the
only ones discussing it.

RE: [PROPOSAL] move away from time-based releases and/or revamp release process

[Animesh Chaturvedi <an...@citrix.com>]

> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Monday, September 23, 2013 12:25 PM
> To: dev@cloudstack.apache.org
> Subject: RE: [PROPOSAL] move away from time-based releases and/or revamp
> release process
> 
> On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi"
> <an...@citrix.com>
> wrote:
> >
> >
> >
> > > -----Original Message-----
> > > From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> > > Sent: Monday, September 23, 2013 11:38 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: [PROPOSAL] move away from time-based releases and/or revamp
> > > release process
> > >
> > > Guys,  I think we are not currently in a state to handle time-based
> > > releases.  Until we can cut master at any time and have it
> > > releasable, or at least at a reasonable RC-level matching minimum
> > > tested requirements, it's just going to continue to be an exercise
> > > in frustration to cut RCs simply because we hit a deadline.
> > [Animesh>] David is going to propose Release Criterion up for
> > discussion
> as per his thread [1]
> 
> I see that thread more about defining what minimum bar we should always
> have master at in order to meet time-based releases. Its where we want
> to go, but not what to do in the meantime.
[Animesh>] His proposal is not just for master, but also for deciding the release exit criterion and IMO is something we should follow for 4.3.0 and onwards
> 
> > >
> > > Maybe we can get away with sticking to time-based if we revamp our
> > > schedule and procedures, I don't know, but in light of how 4.1
> > > (dragged on so long that some were seriously considering
> > > skipping/not releasing it with 4.2 on its heels) and 4.2 (six rounds
> > > of votes so
> > > far) have worked it's probably worth discussing.
> > >
> > > Any suggestions on what might be better? It's been mentioned in the
> > > past that it's a chicken-egg thing, many really don't try it until
> > > we hit an RC, which causes multiple iterations. I do agree that many
> > > don't take it seriously until we start cutting artifacts, but maybe
> > > we do this in a more deliberate fashion instead of jumping right to
> > > the vote. After feature/code freeze, cut some alpha artifacts, wait
> > > a week, cut alpha2 or some beta artifacts, etc, and then at some
> > > point anyone can propose that certain artifacts (or a new set of
> > > artifacts) be put up for a vote as an RC. This gives us a way to
> > > signal that we're gearing up for release and gives plenty of time
> > > for people to test their components, or see the [PROPOSAL] and say
> > > 'oh crap, I had better test my stuff', prior to cutting an RC.
> > > Maybe this wouldn't help in practice, but I think right now we go
> > > from telling the community "code is frozen, don't check anything in
> > > unless its a bug fix" to "here's our RC, try it out", without a
> formal testing window.
> > > I realize the whole thing should be a testing window, but I don't
> > > think it's conveyed well.
> >
> > [Animesh>] After the code freeze is all the stabilization and
> > integration
> testing phase and has been documented at [2].  No one should be waiting
> until the RC to test their components for the first time. It should be
> happening after code freeze.
> 
> >
> > [1] http://markmail.org/thread/wlaq4zg36xnpgsjm
> > [2] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases
> >
> 
> Got it. As mentioned I realize that the whole time there is supposed to
> be testing, but its not really working that way in practice. People are
> volunteers, they forget where things are, or they dont want to mess with
> it unless there is an indication that its semi-stable, and then suddenly
> an RC is thrown over the fence and we go through iterations of RC. By
> the time the RC comes through we should be done testing and just verify
> that someone's last minute bug fix didn't cause a regression or
> something.
[Animesh>] RC is not thrown in it is discussed as part of the release schedule.  After the code-freeze date everyone is expected to complete their integration testing by RC date. In fact I had sent numerous reminders prior to the first RC starting from 2 weeks before the proposed RC date. 

RE: [PROPOSAL] move away from time-based releases and/or revamp release process

[Marcus Sorensen <sh...@gmail.com>]

On Sep 23, 2013 1:03 PM, "Animesh Chaturvedi" <an...@citrix.com>
wrote:
>
>
>
> > -----Original Message-----
> > From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> > Sent: Monday, September 23, 2013 11:38 AM
> > To: dev@cloudstack.apache.org
> > Subject: [PROPOSAL] move away from time-based releases and/or revamp
> > release process
> >
> > Guys,  I think we are not currently in a state to handle time-based
> > releases.  Until we can cut master at any time and have it releasable,
> > or at least at a reasonable RC-level matching minimum tested
> > requirements, it's just going to continue to be an exercise in
> > frustration to cut RCs simply because we hit a deadline.
> [Animesh>] David is going to propose Release Criterion up for discussion
as per his thread [1]

I see that thread more about defining what minimum bar we should always
have master at in order to meet time-based releases. Its where we want to
go, but not what to do in the meantime.

> >
> > Maybe we can get away with sticking to time-based if we revamp our
> > schedule and procedures, I don't know, but in light of how 4.1 (dragged
> > on so long that some were seriously considering skipping/not releasing
> > it with 4.2 on its heels) and 4.2 (six rounds of votes so
> > far) have worked it's probably worth discussing.
> >
> > Any suggestions on what might be better? It's been mentioned in the past
> > that it's a chicken-egg thing, many really don't try it until we hit an
> > RC, which causes multiple iterations. I do agree that many don't take it
> > seriously until we start cutting artifacts, but maybe we do this in a
> > more deliberate fashion instead of jumping right to the vote. After
> > feature/code freeze, cut some alpha artifacts, wait a week, cut alpha2
> > or some beta artifacts, etc, and then at some point anyone can propose
> > that certain artifacts (or a new set of artifacts) be put up for a vote
> > as an RC. This gives us a way to signal that we're gearing up for
> > release and gives plenty of time for people to test their components, or
> > see the [PROPOSAL] and say 'oh crap, I had better test my stuff', prior
> > to cutting an RC.  Maybe this wouldn't help in practice, but I think
> > right now we go from telling the community "code is frozen, don't check
> > anything in unless its a bug fix" to "here's our RC, try it out",
> > without a formal testing window.
> > I realize the whole thing should be a testing window, but I don't think
> > it's conveyed well.
>
> [Animesh>] After the code freeze is all the stabilization and integration
testing phase and has been documented at [2].  No one should be waiting
until the RC to test their components for the first time. It should be
happening after code freeze.

>
> [1] http://markmail.org/thread/wlaq4zg36xnpgsjm
> [2] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases
>

Got it. As mentioned I realize that the whole time there is supposed to be
testing, but its not really working that way in practice. People are
volunteers, they forget where things are, or they dont want to mess with it
unless there is an indication that its semi-stable,
and then suddenly an RC is thrown over the fence and we go through
iterations of RC. By the time the RC comes through we should be done
testing and just verify that someone's last minute bug fix didn't cause a
regression or something.

RE: [PROPOSAL] move away from time-based releases and/or revamp release process

[Animesh Chaturvedi <an...@citrix.com>]

> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: Monday, September 23, 2013 11:38 AM
> To: dev@cloudstack.apache.org
> Subject: [PROPOSAL] move away from time-based releases and/or revamp
> release process
> 
> Guys,  I think we are not currently in a state to handle time-based
> releases.  Until we can cut master at any time and have it releasable,
> or at least at a reasonable RC-level matching minimum tested
> requirements, it's just going to continue to be an exercise in
> frustration to cut RCs simply because we hit a deadline.
[Animesh>] David is going to propose Release Criterion up for discussion as per his thread [1] 
> 
> Maybe we can get away with sticking to time-based if we revamp our
> schedule and procedures, I don't know, but in light of how 4.1 (dragged
> on so long that some were seriously considering skipping/not releasing
> it with 4.2 on its heels) and 4.2 (six rounds of votes so
> far) have worked it's probably worth discussing.
> 
> Any suggestions on what might be better? It's been mentioned in the past
> that it's a chicken-egg thing, many really don't try it until we hit an
> RC, which causes multiple iterations. I do agree that many don't take it
> seriously until we start cutting artifacts, but maybe we do this in a
> more deliberate fashion instead of jumping right to the vote. After
> feature/code freeze, cut some alpha artifacts, wait a week, cut alpha2
> or some beta artifacts, etc, and then at some point anyone can propose
> that certain artifacts (or a new set of artifacts) be put up for a vote
> as an RC. This gives us a way to signal that we're gearing up for
> release and gives plenty of time for people to test their components, or
> see the [PROPOSAL] and say 'oh crap, I had better test my stuff', prior
> to cutting an RC.  Maybe this wouldn't help in practice, but I think
> right now we go from telling the community "code is frozen, don't check
> anything in unless its a bug fix" to "here's our RC, try it out",
> without a formal testing window.
> I realize the whole thing should be a testing window, but I don't think
> it's conveyed well.

[Animesh>] After the code freeze is all the stabilization and integration testing phase and has been documented at [2].  No one should be waiting until the RC to test their components for the first time. It should be happening after code freeze.

[1] http://markmail.org/thread/wlaq4zg36xnpgsjm
[2] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases