You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rene Moser <ma...@renemoser.net> on 2016/01/07 13:39:07 UTC
Minor releases!
Hi
I don't want to be the Grinch but this new release cycle feels not
matching users needs. Especially not my employers one.
We are releasing major versions every month? Great! So we release a
minors every week/day!? No, we don't! :( Every couple of months actually.
A release not getting minors is not a release! It is a dead horse.
Questions:
* Who will use 4.6? when 4.8 was released?
* Is 4.6 still maintained, is 4.5 maintained? is 4.4 maintained?
* Do we really think, clouds will be major upgraded every month?
* If we do a major release every month, why is the last minor 4.5.2 in
August 2015 and not 4.5.3 or 4.5.4?
And if you think there were no commits since 4.5.2?
git rev-list 4.5.2..HEAD --count
77
Branch 4.4:
git rev-list 4.4.4..HEAD --count
8
At the end users are more confused, which version to take. We are
dealing with _infrastructure_.
We must rethink the major release cycling, if it impacts minor releases.
I need a stable cloud aka lots and lots of minors and those now.
Regards
René
Re: Minor releases!
Posted by Ron Wheeler <rw...@artifact-software.com>.
I think that the current numbering scheme blurs the distinction between
"major" and minor".
Without being a developer it is hard to predict the level of risk
involved or the difficulty in reverting or even the amount of downtime
that needs to be allotted to the upgrade.
If the dev team planning and doing the release is careful and makes a
good decision about whether the January release is 4.7.1 or 4.8, that
can send a strong signal to users about the risk of moving forward.
The users are captive to the Cloudstack community's willingness to
support a February release of 4.7.2 and 4.8 if both minor fixes are
backported to 4.7.2 at the same time as a major 4.8 version is released.
Given the fact that this is infrastructure and already very functional,
I suspect that many people would prefer to get the 4.7.2 bug fixes
rather than 4.8 unless they need the new features right away.
They might also prefer to wait to see how the new features in 4.8 are
impacting the early adopters.
I think that the rapid pace of release is probably a good thing overall
since it gets bugs fixed quickly and probably results in a less risky or
time consuming upgrades.
Ron
On 07/01/2016 9:22 AM, Remi Bergsma wrote:
> Hi Erik,
>
>
>
>> And even if 4.6 and 4.7 is considered stable, and testing efforts have
>> increased dramastically, there is always a chance for regressions.
>> Just take a look at all the VR fixes that has come in after 4.6 (I am not
>> saying they are all regressions!)
> That’s a nice example of why we needed to change. The VPC refactor (that happened from 4.5 -> 4.6) introduced problems that we should have seen before (but didn’t). We should make sure that such big changes are broken into smaller pieces and are testable. At least we can fix them quickly now, although the PRs that are open for these issues aren’t being reviewed and tested unfortunately.
>
>
>> I did try a 4.6 upgrade from 4.5 which essentially broke a lot of our VPC
>> usage and had to revert.
>> As a user I have to account for that before doing the upgrade, which means
>> I have to do it in a maintenance window.
>>
>> With minor releases the risk is less, because there's no new features, just
>> bug fixes (and possible improvements?), and that is easier to account for
>> if you don't really need the new features right away.
> That’s a dangerous assumption. We upgraded from 4.4.2 to 4.4.4 and that was hell.
>
>
>> Another note on the rapid release schedule is that it stresses testing
>> efforts, you already scream about more testers, but essentially testing is
>> a 1-week job every month/release, not everyone have time and resources for
>> that.
> I referred to testing Pull Requests, so testing before stuff gets in. If we do that properly, testing the RCs takes little effort.
>
>
>> To summarize; I (and hopefully most other users) really appreciate all the
>> hard work done by SBP, SB, Leasweb, PCextreme +++, but not everyone is in
>> the same positition to do rapid upgrades as you guys are.
> Which means we should stop it? Really, just be clear. It’s perfectly fine with me. Just need to know what people want.
>
> Regards,
> Remi
>
>
>
>
>>
>> Let me know!
>>> Regards,
>>> Remi
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
>>>
>>>
>>> PS: At work we run multiple CloudStack powered clouds with hundreds of
>>> hypervisors so I think I know what I'm talking about.
>>>
>>>
>>>
>> Nobody questioned your skills, but they are not you. And for someone with a
>> single install with 2 hypervisors they don't necessarily have the same
>> amount of resources available.
>>
>> --
>> Erik
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
Re: Minor releases!
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi Erik,
>And even if 4.6 and 4.7 is considered stable, and testing efforts have
>increased dramastically, there is always a chance for regressions.
>Just take a look at all the VR fixes that has come in after 4.6 (I am not
>saying they are all regressions!)
That’s a nice example of why we needed to change. The VPC refactor (that happened from 4.5 -> 4.6) introduced problems that we should have seen before (but didn’t). We should make sure that such big changes are broken into smaller pieces and are testable. At least we can fix them quickly now, although the PRs that are open for these issues aren’t being reviewed and tested unfortunately.
>I did try a 4.6 upgrade from 4.5 which essentially broke a lot of our VPC
>usage and had to revert.
>As a user I have to account for that before doing the upgrade, which means
>I have to do it in a maintenance window.
>
>With minor releases the risk is less, because there's no new features, just
>bug fixes (and possible improvements?), and that is easier to account for
>if you don't really need the new features right away.
That’s a dangerous assumption. We upgraded from 4.4.2 to 4.4.4 and that was hell.
>Another note on the rapid release schedule is that it stresses testing
>efforts, you already scream about more testers, but essentially testing is
>a 1-week job every month/release, not everyone have time and resources for
>that.
I referred to testing Pull Requests, so testing before stuff gets in. If we do that properly, testing the RCs takes little effort.
>To summarize; I (and hopefully most other users) really appreciate all the
>hard work done by SBP, SB, Leasweb, PCextreme +++, but not everyone is in
>the same positition to do rapid upgrades as you guys are.
Which means we should stop it? Really, just be clear. It’s perfectly fine with me. Just need to know what people want.
Regards,
Remi
>
>
>Let me know!
>>
>> Regards,
>> Remi
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
>>
>>
>> PS: At work we run multiple CloudStack powered clouds with hundreds of
>> hypervisors so I think I know what I'm talking about.
>>
>>
>>
>Nobody questioned your skills, but they are not you. And for someone with a
>single install with 2 hypervisors they don't necessarily have the same
>amount of resources available.
>
>--
>Erik
Re: Minor releases!
Posted by Erik Weber <te...@gmail.com>.
On Thu, Jan 7, 2016 at 2:28 PM, Remi Bergsma <RB...@schubergphilis.com>
wrote:
> Hi René, all,
>
> I simply don’t understand why you need lots and lots of minor versions. I
> do understand you need a stable cloud, and that’s exactly what we’re
> achieving here.
>
> We changed our way of working from 4.6 on. Before that, it took _a long_
> time to release new versions (be it major, minor or patch). Releases were
> not high quality so people waited for .1 or .2 to be released. When you did
> eventually upgrade, you’d hit major trouble. It was simply not OK. And many
> minor releases don’t help here. The root cause of the trouble is that the
> whole idea of branching off a new version and have a QA team make it stable
> (while the rest continues on master) doesn’t make sense (any more). That’s
> why in the summer of 2015 we decided to stabilise master instead and
> release from that. [1] One final time it took a great effort to make a
> branch stable.
>
> We released 4.6 in Nov 2015
> We released 4.7 in Dec 2015
> We will release 4.8 in Jan 2016 (unless people think I should stop doing
> this)
>
> In between, we submitted patch releases to quickly address bugs (4.6.1,
> 4.6.2 etc).
>
> You know what? It’s easy to do these upgrades. Much much easier than
> before. The change is small, the procedure is quick. I upgraded our
> production environment from 4.6.2 to 4.7.0 in under 10 minutes (a clustered
> setup). When 4.7.0 got released, for the first time ever, we knew 100% sure
> it included everything from 4.6 as it was built on top of it. No regression
> between 4.6 and 4.7. Actually there’s no need to use 4.6 any more, when 4.7
> is out. It’s essentially the same, plus new features.
>
> So, yes, monthly releases can be done and the quality is better than
> before. Actually, I think we should go much faster. Whenever a PR is merged
> that fixes your issue, it should be possible to deploy it right away. It’s
> a change in mindset.
>
>
> Before this change, master was broken all the time. Now you can even run
> production from master. It just works (tm). Users should pick the latest
> version.
>
> If someone wants to maintain an older release, that can be done. I know
> Rohit maintains older versions for ShapeBlue customers and that makes
> sense. Until everyone gets used to an agile way of working this may still
> be needed. I just don’t work that way any more.
>
>
>
>
> To summarise, you want a stable cloud and I think since 4.6 we provide
> just that. Quick releases, fast delivery of new features and even faster
> delivery of bug fixes. There will always be bugs, so the faster we release,
> the smaller the diff and the easier the deploy and best of all: the sooner
> the bug is resolved in production.
>
> To be honest, I have the feeling not much people agree with me. So maybe I
> should stop doing this. I also see it in the number of people reviewing and
> testing. It’s simply disappointing. Anyway, let me know what you guys want.
> If people want to go back to the old way of working then that’s also fine
> with me. I just won’t be the Release Manager of it.
>
I don't think people necessarily disagree with you, but they have/come from
a different background.
You are heavily involved with the development of CloudStack, a lot of our
users (including me) are not.
And even if 4.6 and 4.7 is considered stable, and testing efforts have
increased dramastically, there is always a chance for regressions.
Just take a look at all the VR fixes that has come in after 4.6 (I am not
saying they are all regressions!)
I did try a 4.6 upgrade from 4.5 which essentially broke a lot of our VPC
usage and had to revert.
As a user I have to account for that before doing the upgrade, which means
I have to do it in a maintenance window.
With minor releases the risk is less, because there's no new features, just
bug fixes (and possible improvements?), and that is easier to account for
if you don't really need the new features right away.
Another note on the rapid release schedule is that it stresses testing
efforts, you already scream about more testers, but essentially testing is
a 1-week job every month/release, not everyone have time and resources for
that.
To summarize; I (and hopefully most other users) really appreciate all the
hard work done by SBP, SB, Leasweb, PCextreme +++, but not everyone is in
the same positition to do rapid upgrades as you guys are.
Let me know!
>
> Regards,
> Remi
>
> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
>
>
> PS: At work we run multiple CloudStack powered clouds with hundreds of
> hypervisors so I think I know what I'm talking about.
>
>
>
Nobody questioned your skills, but they are not you. And for someone with a
single install with 2 hypervisors they don't necessarily have the same
amount of resources available.
--
Erik
Re: Minor releases!
Posted by Daan Hoogland <da...@gmail.com>.
one other remark: no matter what our release scheme, we need for every one
to do twice as much reviews as contributions in order for our present way
of working to be successful. And in this doing a review means sticking with
a PR until it is good enough to go in, not just make initial remarks. We
have close to 200 open PRs now and it has not been getting better since we
moved to github. I am starting to feel this move was no good.
Not really related but: We can not easily close PRs or merge them wich is
only a hassle and not stopping us but it's a bother to the RM.
On Thu, Jan 7, 2016 at 10:30 PM, Marcus <sh...@gmail.com> wrote:
> A few things to note:
>
> 1. I repeated this until I felt bad about harping on it, but we were seeing
> new functionality slip into minors all the time. The idea that 4.4.x ->
> 4.4.y upgrade was safer than 4.4.x -> 4.5.0 (just made up versions as an
> example) is unfortunately wrong.
>
> 2. I agree that large production environments will probably only want to
> upgrade their cloud once every few months, at the most. Probably less, due
> to internal testing and qualification for each chosen major. I understand
> the mindset someone might have when they're told that they need to jump six
> major releases and pull in dozens of new features to fix one or two bugs
> they have come across in their otherwise stable cloud. Suddenly they're
> thinking they need to do far more integration testing with their internal
> CloudStack consumers and their infrastructure. Even if the releases are
> more stable, any major release is going to be treated as an unknown and
> require rigorous qualifications. Due to #1, this point is largely moot,
> but if we were doing proper bugfix releases, or chose to go that direction,
> it might be something important to think about.
>
> In the end, it probably doesn't matter much either way for routine
> maintenance. If you were upgrading once every 6-12 months to keep up with
> the previous major cycle, you can still upgrade at that pace and the delta
> in changes between your chosen minors will be roughly what it was before,
> even if the minors aren't contiguous numbers. The place this breaks down is
> that while you can pick a release from any month and decide to use that as
> your next 6-12 month version, there are far fewer people running your
> version or focusing on it for bugfixes; you've been left behind once you've
> settled on running that version. So you either have to get with the program
> and figure out how to do rapid releases in your environment that is averse
> to change and full of red tape, or start maintaining your own release
> builds. That's kind of a crappy choice that we're putting on our less agile
> users.
>
> To that end, I like the idea of the LTS, simply tagging majors as the
> preferred version for people who require a slower upgrade. This would focus
> those users into certain versions and improve maintenance on them. It could
> even just be unofficial, a version that CloudStack users agree on. From a
> versioning perspective this doesn't sound a whole lot different from the
> old scheme where 4.4 and 4.5 were essentially an LTS, but it is still
> better because we've improved the release process. Of course, we still have
> to reign in the changes to these LTS versions and stop allowing new
> features to creep in simply because customers who are on that LTS need the
> feature.
>
> On Thu, Jan 7, 2016 at 12:30 PM Wei ZHOU <us...@gmail.com> wrote:
>
> > 2016-01-07 15:24 GMT+01:00 Rene Moser <ma...@renemoser.net>:
> >
> > > Hi Remi
> > >
> > > On 01/07/2016 02:28 PM, Remi Bergsma wrote:
> > > > I simply don’t understand why you need lots and lots of minor
> versions.
> > > I do understand you need a stable cloud, and that’s exactly what we’re
> > > achieving here.
> > > >
> > > > We changed our way of working from 4.6 on. Before that, it took _a
> > long_
> > > time to release new versions (be it major, minor or patch). Releases
> were
> > > not high quality so people waited for .1 or .2 to be released. When you
> > did
> > > eventually upgrade, you’d hit major trouble. It was simply not OK. And
> > many
> > > minor releases don’t help here. The root cause of the trouble is that
> the
> > > whole idea of branching off a new version and have a QA team make it
> > stable
> > > (while the rest continues on master) doesn’t make sense (any more).
> > That’s
> > > why in the summer of 2015 we decided to stabilise master instead and
> > > release from that. [1] One final time it took a great effort to make a
> > > branch stable.
> > > >
> > > > We released 4.6 in Nov 2015
> > > > We released 4.7 in Dec 2015
> > > > We will release 4.8 in Jan 2016 (unless people think I should stop
> > doing
> > > this)
> > > >
> > > > In between, we submitted patch releases to quickly address bugs
> (4.6.1,
> > > 4.6.2 etc).
> > >
> > > What about 4.5? 4.4? Is 4.6 still getting 4.6.3? Users don't see that.
> > >
> > > > You know what? It’s easy to do these upgrades. Much much easier than
> > > before. The change is small, the procedure is quick.
> > >
> > > We are not yet on 4.6, so we have downtime by upgrading the routers, we
> > > have approx 120 VR, which take about ~ 5-10 minutes of downtime each.
> > >
> > >
> > If you use isolated network with redundant VR, you can try to apply the
> PR
> > https://github.com/apache/cloudstack/pull/1198
> > then restart networks with cleanup, the new VRs will run with new
> systemvm
> > template, downtime is acceptable (1s-3s).
> >
> > We used isolated network with single VR before, we had another patch to
> > modify the networks to be with RVR, and the existing VR as master VR.
> > However, the patch is not applicable for 4.6 and later.
> >
> >
> > > We are not agile in upgrading, we have freeze times and plan our
> > > releases and test each release intensively (it saved our asses in the
> > > past).
> > >
> > > Every bigger software project in the world has agreed that providing
> > > minor releases is good practice:
> > >
> > > - New features --> Major release
> > > - Fixes --> Minor release
> > >
> > > The problem with new features are potentially new bugs (by nature):
> > >
> > > Now you are basically saying:
> > >
> > > - New features and fixes --> Major release only
> > >
> > > By definition, you will end up it a more unstable software because you
> > > will only get fixes in cost of new features and potentially new bugs. I
> > > don't call this stable.
> > >
> > > I do not see any benefit of the current release rush.
> > >
> > > René
> > >
> >
>
--
Daan
Re: Minor releases!
Posted by Daan Hoogland <da...@gmail.com>.
I totally agree Marcus, with one small addition. In our present scheme we
can mark any 4.x as LTS if we maintain the discipline of fixing any bug on
the oldest version known to contain the bug and merge the fix forward. some
LIMITATION OF WARRANTY (tm) apply of course; if a fix requires some kind
of database change...
Basically our present release scheme was designed to address the concerns
you are describing.
On Thu, Jan 7, 2016 at 10:30 PM, Marcus <sh...@gmail.com> wrote:
> A few things to note:
>
> 1. I repeated this until I felt bad about harping on it, but we were seeing
> new functionality slip into minors all the time. The idea that 4.4.x ->
> 4.4.y upgrade was safer than 4.4.x -> 4.5.0 (just made up versions as an
> example) is unfortunately wrong.
>
> 2. I agree that large production environments will probably only want to
> upgrade their cloud once every few months, at the most. Probably less, due
> to internal testing and qualification for each chosen major. I understand
> the mindset someone might have when they're told that they need to jump six
> major releases and pull in dozens of new features to fix one or two bugs
> they have come across in their otherwise stable cloud. Suddenly they're
> thinking they need to do far more integration testing with their internal
> CloudStack consumers and their infrastructure. Even if the releases are
> more stable, any major release is going to be treated as an unknown and
> require rigorous qualifications. Due to #1, this point is largely moot,
> but if we were doing proper bugfix releases, or chose to go that direction,
> it might be something important to think about.
>
> In the end, it probably doesn't matter much either way for routine
> maintenance. If you were upgrading once every 6-12 months to keep up with
> the previous major cycle, you can still upgrade at that pace and the delta
> in changes between your chosen minors will be roughly what it was before,
> even if the minors aren't contiguous numbers. The place this breaks down is
> that while you can pick a release from any month and decide to use that as
> your next 6-12 month version, there are far fewer people running your
> version or focusing on it for bugfixes; you've been left behind once you've
> settled on running that version. So you either have to get with the program
> and figure out how to do rapid releases in your environment that is averse
> to change and full of red tape, or start maintaining your own release
> builds. That's kind of a crappy choice that we're putting on our less agile
> users.
>
> To that end, I like the idea of the LTS, simply tagging majors as the
> preferred version for people who require a slower upgrade. This would focus
> those users into certain versions and improve maintenance on them. It could
> even just be unofficial, a version that CloudStack users agree on. From a
> versioning perspective this doesn't sound a whole lot different from the
> old scheme where 4.4 and 4.5 were essentially an LTS, but it is still
> better because we've improved the release process. Of course, we still have
> to reign in the changes to these LTS versions and stop allowing new
> features to creep in simply because customers who are on that LTS need the
> feature.
>
--
Daan
Re: Minor releases!
Posted by Marcus <sh...@gmail.com>.
A few things to note:
1. I repeated this until I felt bad about harping on it, but we were seeing
new functionality slip into minors all the time. The idea that 4.4.x ->
4.4.y upgrade was safer than 4.4.x -> 4.5.0 (just made up versions as an
example) is unfortunately wrong.
2. I agree that large production environments will probably only want to
upgrade their cloud once every few months, at the most. Probably less, due
to internal testing and qualification for each chosen major. I understand
the mindset someone might have when they're told that they need to jump six
major releases and pull in dozens of new features to fix one or two bugs
they have come across in their otherwise stable cloud. Suddenly they're
thinking they need to do far more integration testing with their internal
CloudStack consumers and their infrastructure. Even if the releases are
more stable, any major release is going to be treated as an unknown and
require rigorous qualifications. Due to #1, this point is largely moot,
but if we were doing proper bugfix releases, or chose to go that direction,
it might be something important to think about.
In the end, it probably doesn't matter much either way for routine
maintenance. If you were upgrading once every 6-12 months to keep up with
the previous major cycle, you can still upgrade at that pace and the delta
in changes between your chosen minors will be roughly what it was before,
even if the minors aren't contiguous numbers. The place this breaks down is
that while you can pick a release from any month and decide to use that as
your next 6-12 month version, there are far fewer people running your
version or focusing on it for bugfixes; you've been left behind once you've
settled on running that version. So you either have to get with the program
and figure out how to do rapid releases in your environment that is averse
to change and full of red tape, or start maintaining your own release
builds. That's kind of a crappy choice that we're putting on our less agile
users.
To that end, I like the idea of the LTS, simply tagging majors as the
preferred version for people who require a slower upgrade. This would focus
those users into certain versions and improve maintenance on them. It could
even just be unofficial, a version that CloudStack users agree on. From a
versioning perspective this doesn't sound a whole lot different from the
old scheme where 4.4 and 4.5 were essentially an LTS, but it is still
better because we've improved the release process. Of course, we still have
to reign in the changes to these LTS versions and stop allowing new
features to creep in simply because customers who are on that LTS need the
feature.
On Thu, Jan 7, 2016 at 12:30 PM Wei ZHOU <us...@gmail.com> wrote:
> 2016-01-07 15:24 GMT+01:00 Rene Moser <ma...@renemoser.net>:
>
> > Hi Remi
> >
> > On 01/07/2016 02:28 PM, Remi Bergsma wrote:
> > > I simply don’t understand why you need lots and lots of minor versions.
> > I do understand you need a stable cloud, and that’s exactly what we’re
> > achieving here.
> > >
> > > We changed our way of working from 4.6 on. Before that, it took _a
> long_
> > time to release new versions (be it major, minor or patch). Releases were
> > not high quality so people waited for .1 or .2 to be released. When you
> did
> > eventually upgrade, you’d hit major trouble. It was simply not OK. And
> many
> > minor releases don’t help here. The root cause of the trouble is that the
> > whole idea of branching off a new version and have a QA team make it
> stable
> > (while the rest continues on master) doesn’t make sense (any more).
> That’s
> > why in the summer of 2015 we decided to stabilise master instead and
> > release from that. [1] One final time it took a great effort to make a
> > branch stable.
> > >
> > > We released 4.6 in Nov 2015
> > > We released 4.7 in Dec 2015
> > > We will release 4.8 in Jan 2016 (unless people think I should stop
> doing
> > this)
> > >
> > > In between, we submitted patch releases to quickly address bugs (4.6.1,
> > 4.6.2 etc).
> >
> > What about 4.5? 4.4? Is 4.6 still getting 4.6.3? Users don't see that.
> >
> > > You know what? It’s easy to do these upgrades. Much much easier than
> > before. The change is small, the procedure is quick.
> >
> > We are not yet on 4.6, so we have downtime by upgrading the routers, we
> > have approx 120 VR, which take about ~ 5-10 minutes of downtime each.
> >
> >
> If you use isolated network with redundant VR, you can try to apply the PR
> https://github.com/apache/cloudstack/pull/1198
> then restart networks with cleanup, the new VRs will run with new systemvm
> template, downtime is acceptable (1s-3s).
>
> We used isolated network with single VR before, we had another patch to
> modify the networks to be with RVR, and the existing VR as master VR.
> However, the patch is not applicable for 4.6 and later.
>
>
> > We are not agile in upgrading, we have freeze times and plan our
> > releases and test each release intensively (it saved our asses in the
> > past).
> >
> > Every bigger software project in the world has agreed that providing
> > minor releases is good practice:
> >
> > - New features --> Major release
> > - Fixes --> Minor release
> >
> > The problem with new features are potentially new bugs (by nature):
> >
> > Now you are basically saying:
> >
> > - New features and fixes --> Major release only
> >
> > By definition, you will end up it a more unstable software because you
> > will only get fixes in cost of new features and potentially new bugs. I
> > don't call this stable.
> >
> > I do not see any benefit of the current release rush.
> >
> > René
> >
>
Re: Minor releases!
Posted by Wei ZHOU <us...@gmail.com>.
2016-01-07 15:24 GMT+01:00 Rene Moser <ma...@renemoser.net>:
> Hi Remi
>
> On 01/07/2016 02:28 PM, Remi Bergsma wrote:
> > I simply don’t understand why you need lots and lots of minor versions.
> I do understand you need a stable cloud, and that’s exactly what we’re
> achieving here.
> >
> > We changed our way of working from 4.6 on. Before that, it took _a long_
> time to release new versions (be it major, minor or patch). Releases were
> not high quality so people waited for .1 or .2 to be released. When you did
> eventually upgrade, you’d hit major trouble. It was simply not OK. And many
> minor releases don’t help here. The root cause of the trouble is that the
> whole idea of branching off a new version and have a QA team make it stable
> (while the rest continues on master) doesn’t make sense (any more). That’s
> why in the summer of 2015 we decided to stabilise master instead and
> release from that. [1] One final time it took a great effort to make a
> branch stable.
> >
> > We released 4.6 in Nov 2015
> > We released 4.7 in Dec 2015
> > We will release 4.8 in Jan 2016 (unless people think I should stop doing
> this)
> >
> > In between, we submitted patch releases to quickly address bugs (4.6.1,
> 4.6.2 etc).
>
> What about 4.5? 4.4? Is 4.6 still getting 4.6.3? Users don't see that.
>
> > You know what? It’s easy to do these upgrades. Much much easier than
> before. The change is small, the procedure is quick.
>
> We are not yet on 4.6, so we have downtime by upgrading the routers, we
> have approx 120 VR, which take about ~ 5-10 minutes of downtime each.
>
>
If you use isolated network with redundant VR, you can try to apply the PR
https://github.com/apache/cloudstack/pull/1198
then restart networks with cleanup, the new VRs will run with new systemvm
template, downtime is acceptable (1s-3s).
We used isolated network with single VR before, we had another patch to
modify the networks to be with RVR, and the existing VR as master VR.
However, the patch is not applicable for 4.6 and later.
> We are not agile in upgrading, we have freeze times and plan our
> releases and test each release intensively (it saved our asses in the
> past).
>
> Every bigger software project in the world has agreed that providing
> minor releases is good practice:
>
> - New features --> Major release
> - Fixes --> Minor release
>
> The problem with new features are potentially new bugs (by nature):
>
> Now you are basically saying:
>
> - New features and fixes --> Major release only
>
> By definition, you will end up it a more unstable software because you
> will only get fixes in cost of new features and potentially new bugs. I
> don't call this stable.
>
> I do not see any benefit of the current release rush.
>
> René
>
Re: Minor releases!
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Heaven is closer than you might think.
4.7 did not require a systemvm template change, so we reused the existing one. No impact in upgrading. We need to change it only when needed, it’s that simple.
A separate project and version is probably best way forward.
Regards,
Remi
On 07/01/16 15:50, "Wido den Hollander" <wi...@widodh.nl> wrote:
>
>
>On 01/07/2016 03:47 PM, Nux! wrote:
>>> And that is indeed it. I'm not keen on doing point releases either.
>>
>>> Simplify upgrade paths, ditch the fact that a VR has to be upgraded
>>> every time, etc, etc.
>>
>> That'd be heaven. I hate the VR upgrades!
>>
>
>A version might have a flag telling if the VR needs upgrade or not.
>Boolean, true/false.
>
>>>
>>> We could also call 4.8 simply 4.7.2. It's just a number :)
>>
>> +1, perhaps switch to version based on dates?
>>
>
>No, my point is. Just the fact that it is called 4.8 makes people itchy.
>I really wonder why.
>
>We could also call it 5.0 and that will keep people away from it?
>
>Wido
>
>>>
>>> Wido
>>
>>
Re: Minor releases!
Posted by sebgoa <ru...@gmail.com>.
>> We can not even tell users which releases are still maintained and
>> getting fixes, even how long the latest release gets fixes...
>
> Why can’t we tell them? Because we go faster now? I’m sure we can explain it.
>
I do think we are in a bit of temporary "murky" zone.
We have folks still on 4.4 (and most likely much further), Rohit is still maintaing 4.5.x
and now we have 4.6, 4.6.2, 4.7.0 and 4.8.0 is coming.
I think we should write a "release strategy" and even make a PR about it.
Ultimately everyone needs to upgrade to 4.6.0 and then it *should* be continuous/linear upgrade, because every commit that might go into a minor is in the next major.
and FWIW, Kubernetes which I monitor closely, releases on github minors almost daily and yes it's tough to keep up.
As a matter of fact I think every couple commits there is a minor release.
> Regards,
> Remi
>
Re: Minor releases!
Posted by Ron Wheeler <rw...@artifact-software.com>.
From a system admin position there is a big difference between 4.7.0
and 4.6.2
Normally 4.6.2 is a minor release with no new features, no changes to
data structures, no changes required to installation prerequisites or
procedures unless the RM writes a big note.
Normally, 4.7.0 is a major release possibly with new features that have
not been tried in a wide variety of hardware and software configurations
even if it has been tested by several developers on their test setups.
It may have hidden gotchas which may only show up in large scale
deployment. I may need to do extensive testing before deploying.
I will generally not run any x.x.0 in production unless I am absolutely
forced to or I don't care about reliability.
I am leery about using a x.x.0 version of a software library in
development unless I am sure that we will be able to detect any changes
while we are testing our software.
We might even release a version of our software with just the library
change to see if it still works.
I understand that the numbering is out of my control but it is a signal
from the people who are smarter than me and are intimately involved with
the development about the risk involved in deploying the software.
It also gives a hint about how carefully the release notes need to be
read. In major releases, there will be some reading between the lines
required to figure out if the new feature will break something that we
have done or is different from the environment that the developers use
for testing.
CentOS 7 broke an amazing number of our internal practices!!! 6.4 to 6.5
was transparent.
The differentiation of major and minor releases is also a signal to the
Cloudstack dev community about what can be attempted in a release.
If you are working on 4.6.3, you should not be thinking about changing
the data layer but in a 4.8, that is a valid topic to bring up.
If you do not have major and minor releases, I do not know what would be
the best way to signal that 4.7.0 is really 4.6.2 but 4.8.0 is not the
same as a 4.6.3.
Ron
On 07/01/2016 11:55 AM, Jeff Moody wrote:
> While the pretty pictures iare helpful, it's explaining the git
> merging/branching strategy which if I were to show to a non-
> developmentally aware coworker, they may be overwhelmed or confused by.
>
> I was unaware that 4.7.0 == 4.6.2 and try and follow development pretty
> actively.
> I also think that while the new release strategy is awesome and _I_
> love the faster releases, making this change halfway through the 4.x
> release cycle is also confusing.
>
> It might be worthwhile to make 4.8 or 4.9 be ACS 5.0 and as a part of
> the release notes make a less-technical overview of how the release
> process has changed and how releases are handled moving forward as,
> while 5.0 may not be a major feature release (following semver) it is a
> radical shift in how ACS is released to the public.
>
> For the pretty picture, I think a graphic showing the previous release
> process of branches off of branches off of branches would help provide
> a valuable comparison as to why the current process is better and more
> reliable than the previous process.
>
>
> On Thu, 2016-01-07 at 16:39 +0000, Remi Bergsma wrote:
>> This page has pretty pictures:
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+princi
>> ples+for+Apache+CloudStack+4.6+and+up
>>
>>
>>
>>
>> On 07/01/16 17:37, "sebgoa" <ru...@gmail.com> wrote:
>>
>>> On Jan 7, 2016, at 5:27 PM, Remi Bergsma <RBergsma@schubergphilis.c
>>> om> wrote:
>>>
>>>> On 07/01/16 17:22, "Rene Moser" <ma...@renemoser.net> wrote:
>>>>> No, it is not the pace. You can do as many major as often as
>>>>> you want
>>>>> but if one uses this major, how long will it get minors? We
>>>>> have no clue.
>>>>>
>>>>> I understand your point completely while my argument is that I
>>>>> have to
>>>>> plan releases year by year.
>>>>>
>>>>> Under this condition I'd take the LTS of the releases, the most
>>>>> stable
>>>>> one even its 2 years old, (I have to maintain it for a year),
>>>>> not the
>>>>> latest one, for sure not a .0 release.
>>>>>
>>>>> With that mindset, there is no version for me right now.
>>>> I see your point. To me this is not a sustainable model, but if
>>>> you want to keep doing this the only option I see is finding a RM
>>>> for your specific release.
>>>>
>>>> And as a matter of fact, 4.7.0 == 4.6.2 so that .0 might be the
>>>> best .0 release we ever had. Don’t underestimate the change that
>>>> was made. Releases now build on top of each other, while that was
>>>> never the case.
>>>>
>>>> Anyway, I cannot and don’t want to convince you. We want
>>>> something different and that is fine. What I do want to know is
>>>> what others want. Because if the majority wants what you are
>>>> asking for, we should do that.
>>>>
>>> Remi, I think Rene might have a point, that while things are clear
>>> for you, the fact that 4.7.0 == 4.6.2 may be lost on other folks
>>> that are used to the old ways.
>>>
>>> Maybe we need a picture or something...
>>>
>>>> Regards,
>>>> Remi
--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com
skype: ronaldmwheeler
phone: 866-970-2435, ext 102
Re: Minor releases!
Posted by Jeff Moody <je...@fifthecho.com>.
While the pretty pictures iare helpful, it's explaining the git
merging/branching strategy which if I were to show to a non-
developmentally aware coworker, they may be overwhelmed or confused by.
I was unaware that 4.7.0 == 4.6.2 and try and follow development pretty
actively.
I also think that while the new release strategy is awesome and _I_
love the faster releases, making this change halfway through the 4.x
release cycle is also confusing.
It might be worthwhile to make 4.8 or 4.9 be ACS 5.0 and as a part of
the release notes make a less-technical overview of how the release
process has changed and how releases are handled moving forward as,
while 5.0 may not be a major feature release (following semver) it is a
radical shift in how ACS is released to the public.
For the pretty picture, I think a graphic showing the previous release
process of branches off of branches off of branches would help provide
a valuable comparison as to why the current process is better and more
reliable than the previous process.
On Thu, 2016-01-07 at 16:39 +0000, Remi Bergsma wrote:
> This page has pretty pictures:
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+princi
> ples+for+Apache+CloudStack+4.6+and+up
>
>
>
>
> On 07/01/16 17:37, "sebgoa" <ru...@gmail.com> wrote:
>
> >
> > On Jan 7, 2016, at 5:27 PM, Remi Bergsma <RBergsma@schubergphilis.c
> > om> wrote:
> >
> > >
> > > On 07/01/16 17:22, "Rene Moser" <ma...@renemoser.net> wrote:
> > > > No, it is not the pace. You can do as many major as often as
> > > > you want
> > > > but if one uses this major, how long will it get minors? We
> > > > have no clue.
> > > >
> > > > I understand your point completely while my argument is that I
> > > > have to
> > > > plan releases year by year.
> > > >
> > > > Under this condition I'd take the LTS of the releases, the most
> > > > stable
> > > > one even its 2 years old, (I have to maintain it for a year),
> > > > not the
> > > > latest one, for sure not a .0 release.
> > > >
> > > > With that mindset, there is no version for me right now.
> > >
> > > I see your point. To me this is not a sustainable model, but if
> > > you want to keep doing this the only option I see is finding a RM
> > > for your specific release.
> > >
> > > And as a matter of fact, 4.7.0 == 4.6.2 so that .0 might be the
> > > best .0 release we ever had. Don’t underestimate the change that
> > > was made. Releases now build on top of each other, while that was
> > > never the case.
> > >
> > > Anyway, I cannot and don’t want to convince you. We want
> > > something different and that is fine. What I do want to know is
> > > what others want. Because if the majority wants what you are
> > > asking for, we should do that.
> > >
> >
> > Remi, I think Rene might have a point, that while things are clear
> > for you, the fact that 4.7.0 == 4.6.2 may be lost on other folks
> > that are used to the old ways.
> >
> > Maybe we need a picture or something...
> >
> > > Regards,
> > > Remi
> >
Re: Minor releases!
Posted by Remi Bergsma <RB...@schubergphilis.com>.
This page has pretty pictures:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
On 07/01/16 17:37, "sebgoa" <ru...@gmail.com> wrote:
>
>On Jan 7, 2016, at 5:27 PM, Remi Bergsma <RB...@schubergphilis.com> wrote:
>
>>
>> On 07/01/16 17:22, "Rene Moser" <ma...@renemoser.net> wrote:
>>> No, it is not the pace. You can do as many major as often as you want
>>> but if one uses this major, how long will it get minors? We have no clue.
>>>
>>> I understand your point completely while my argument is that I have to
>>> plan releases year by year.
>>>
>>> Under this condition I'd take the LTS of the releases, the most stable
>>> one even its 2 years old, (I have to maintain it for a year), not the
>>> latest one, for sure not a .0 release.
>>>
>>> With that mindset, there is no version for me right now.
>>
>> I see your point. To me this is not a sustainable model, but if you want to keep doing this the only option I see is finding a RM for your specific release.
>>
>> And as a matter of fact, 4.7.0 == 4.6.2 so that .0 might be the best .0 release we ever had. Don’t underestimate the change that was made. Releases now build on top of each other, while that was never the case.
>>
>> Anyway, I cannot and don’t want to convince you. We want something different and that is fine. What I do want to know is what others want. Because if the majority wants what you are asking for, we should do that.
>>
>
>Remi, I think Rene might have a point, that while things are clear for you, the fact that 4.7.0 == 4.6.2 may be lost on other folks that are used to the old ways.
>
>Maybe we need a picture or something...
>
>> Regards,
>> Remi
>
Re: Minor releases!
Posted by sebgoa <ru...@gmail.com>.
On Jan 7, 2016, at 5:27 PM, Remi Bergsma <RB...@schubergphilis.com> wrote:
>
> On 07/01/16 17:22, "Rene Moser" <ma...@renemoser.net> wrote:
>> No, it is not the pace. You can do as many major as often as you want
>> but if one uses this major, how long will it get minors? We have no clue.
>>
>> I understand your point completely while my argument is that I have to
>> plan releases year by year.
>>
>> Under this condition I'd take the LTS of the releases, the most stable
>> one even its 2 years old, (I have to maintain it for a year), not the
>> latest one, for sure not a .0 release.
>>
>> With that mindset, there is no version for me right now.
>
> I see your point. To me this is not a sustainable model, but if you want to keep doing this the only option I see is finding a RM for your specific release.
>
> And as a matter of fact, 4.7.0 == 4.6.2 so that .0 might be the best .0 release we ever had. Don’t underestimate the change that was made. Releases now build on top of each other, while that was never the case.
>
> Anyway, I cannot and don’t want to convince you. We want something different and that is fine. What I do want to know is what others want. Because if the majority wants what you are asking for, we should do that.
>
Remi, I think Rene might have a point, that while things are clear for you, the fact that 4.7.0 == 4.6.2 may be lost on other folks that are used to the old ways.
Maybe we need a picture or something...
> Regards,
> Remi
Re: Minor releases!
Posted by Rene Moser <ma...@renemoser.net>.
On 01/07/2016 05:27 PM, Remi Bergsma wrote:
> Anyway, I cannot and don’t want to convince you. We want something different and that is fine. What I do want to know is what others want. Because if the majority wants what you are asking for, we should do that.
It is not my decision, it is a condition.
In a perfect world we could support both uses cases.
The more I think about it, the more I think a CloudStack LTS version
would be what I need.
Thanks again and don't feel offended. I was not my intention.
Kind Regards
René
Re: Minor releases!
Posted by Remi Bergsma <RB...@schubergphilis.com>.
On 07/01/16 17:22, "Rene Moser" <ma...@renemoser.net> wrote:
>No, it is not the pace. You can do as many major as often as you want
>but if one uses this major, how long will it get minors? We have no clue.
>
>I understand your point completely while my argument is that I have to
>plan releases year by year.
>
>Under this condition I'd take the LTS of the releases, the most stable
>one even its 2 years old, (I have to maintain it for a year), not the
>latest one, for sure not a .0 release.
>
>With that mindset, there is no version for me right now.
I see your point. To me this is not a sustainable model, but if you want to keep doing this the only option I see is finding a RM for your specific release.
And as a matter of fact, 4.7.0 == 4.6.2 so that .0 might be the best .0 release we ever had. Don’t underestimate the change that was made. Releases now build on top of each other, while that was never the case.
Anyway, I cannot and don’t want to convince you. We want something different and that is fine. What I do want to know is what others want. Because if the majority wants what you are asking for, we should do that.
Regards,
Remi
Re: Minor releases!
Posted by Rene Moser <ma...@renemoser.net>.
On 01/07/2016 05:08 PM, Remi Bergsma wrote:
>> Good plan but this discussion is about _fixes_ for things we already run
>> on hardware in production in real world conditions. For problems we
>> worked around. For issues we deal with daily and waiting for a release
>> that fixes these.
>
> Sorry, I don’t get your point. I run both 4.4 and 4.7 in production and fast releases help me deploy fixes soon and make users happy.
By users I meant cloudsack users not end users.
>>> Like I said before, if you want to stick on another version, please go ahead and release any version you need. I’m sure Daan, Rohit or myself are more than happy to show you how to do that.
>>
>> It is not about me, it is about reliable releases.
>
> First it was about the pace, now it’s about their reliability?
No, it is not the pace. You can do as many major as often as you want
but if one uses this major, how long will it get minors? We have no clue.
I understand your point completely while my argument is that I have to
plan releases year by year.
Under this condition I'd take the LTS of the releases, the most stable
one even its 2 years old, (I have to maintain it for a year), not the
latest one, for sure not a .0 release.
With that mindset, there is no version for me right now.
Re: Minor releases!
Posted by Remi Bergsma <RB...@schubergphilis.com>.
On 07/01/16 16:33, "Rene Moser" <ma...@renemoser.net> wrote:
>On 01/07/2016 04:10 PM, Remi Bergsma wrote:
>> Even minor releases had issues so I don’t see the difference.
>
>Yes, that is why you make them as small as possible. and they are used
>to improve things.
Which is what we are doing?
>> To me there’s only one road to success: test every PR, run integration tests against real hardware and have someone review the code. Then release as fast as possible.
>> Unfortunately, without write access to Github we will not be able to automate this.
>
>Good plan but this discussion is about _fixes_ for things we already run
>on hardware in production in real world conditions. For problems we
>worked around. For issues we deal with daily and waiting for a release
>that fixes these.
Sorry, I don’t get your point. I run both 4.4 and 4.7 in production and fast releases help me deploy fixes soon and make users happy.
>> Like I said before, if you want to stick on another version, please go ahead and release any version you need. I’m sure Daan, Rohit or myself are more than happy to show you how to do that.
>
>It is not about me, it is about reliable releases.
First it was about the pace, now it’s about their reliability?
>We can not even tell users which releases are still maintained and
>getting fixes, even how long the latest release gets fixes...
Why can’t we tell them? Because we go faster now? I’m sure we can explain it.
Regards,
Remi
Re: Minor releases!
Posted by Rene Moser <ma...@renemoser.net>.
On 01/07/2016 04:10 PM, Remi Bergsma wrote:
> Even minor releases had issues so I don’t see the difference.
Yes, that is why you make them as small as possible. and they are used
to improve things.
> To me there’s only one road to success: test every PR, run integration tests against real hardware and have someone review the code. Then release as fast as possible.
> Unfortunately, without write access to Github we will not be able to automate this.
Good plan but this discussion is about _fixes_ for things we already run
on hardware in production in real world conditions. For problems we
worked around. For issues we deal with daily and waiting for a release
that fixes these.
> Like I said before, if you want to stick on another version, please go ahead and release any version you need. I’m sure Daan, Rohit or myself are more than happy to show you how to do that.
It is not about me, it is about reliable releases.
We can not even tell users which releases are still maintained and
getting fixes, even how long the latest release gets fixes...
Re: Minor releases!
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Even minor releases had issues so I don’t see the difference.
To me there’s only one road to success: test every PR, run integration tests against real hardware and have someone review the code. Then release as fast as possible.
Unfortunately, without write access to Github we will not be able to automate this.
Like I said before, if you want to stick on another version, please go ahead and release any version you need. I’m sure Daan, Rohit or myself are more than happy to show you how to do that.
Regards,
Remi
>Because with the current versioining scheme a new major release means new
>features/big changes = bugs = downtime.
>Atleast that is the assumption based on previous experience.
>
>More or less every major release before 4.6 had issues related to upgrades,
>so there is your reason for why people are itchy.
>Even 4.6.0 had issues after all.
>
>--
>Erik
Re: Minor releases!
Posted by Wido den Hollander <wi...@widodh.nl>.
On 01/07/2016 03:53 PM, Erik Weber wrote:
> On Thu, Jan 7, 2016 at 3:50 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>
>>
>>
>> On 01/07/2016 03:47 PM, Nux! wrote:
>>>> And that is indeed it. I'm not keen on doing point releases either.
>>>
>>>> Simplify upgrade paths, ditch the fact that a VR has to be upgraded
>>>> every time, etc, etc.
>>>
>>> That'd be heaven. I hate the VR upgrades!
>>>
>>
>> A version might have a flag telling if the VR needs upgrade or not.
>> Boolean, true/false.
>>
>>>>
>>>> We could also call 4.8 simply 4.7.2. It's just a number :)
>>>
>>> +1, perhaps switch to version based on dates?
>>>
>>
>> No, my point is. Just the fact that it is called 4.8 makes people itchy.
>> I really wonder why.
>>
>
>
> Because with the current versioining scheme a new major release means new
> features/big changes = bugs = downtime.
> Atleast that is the assumption based on previous experience.
>
> More or less every major release before 4.6 had issues related to upgrades,
> so there is your reason for why people are itchy.
> Even 4.6.0 had issues after all.
>
But we do not fix that by having minor releases. We fix that by writing
proper code and testing it.
Currently a lot of stuff breaks because tests are not fully there. It
could be that somebody fixes A but breaks B without knowing it.
Minor versions actually only slow stuff down and are harder to maintain
imho. Having short release cycles which are stable are a lot easier.
Wido
Re: Minor releases!
Posted by Erik Weber <te...@gmail.com>.
On Thu, Jan 7, 2016 at 3:50 PM, Wido den Hollander <wi...@widodh.nl> wrote:
>
>
> On 01/07/2016 03:47 PM, Nux! wrote:
> >> And that is indeed it. I'm not keen on doing point releases either.
> >
> >> Simplify upgrade paths, ditch the fact that a VR has to be upgraded
> >> every time, etc, etc.
> >
> > That'd be heaven. I hate the VR upgrades!
> >
>
> A version might have a flag telling if the VR needs upgrade or not.
> Boolean, true/false.
>
> >>
> >> We could also call 4.8 simply 4.7.2. It's just a number :)
> >
> > +1, perhaps switch to version based on dates?
> >
>
> No, my point is. Just the fact that it is called 4.8 makes people itchy.
> I really wonder why.
>
Because with the current versioining scheme a new major release means new
features/big changes = bugs = downtime.
Atleast that is the assumption based on previous experience.
More or less every major release before 4.6 had issues related to upgrades,
so there is your reason for why people are itchy.
Even 4.6.0 had issues after all.
--
Erik
Re: Minor releases!
Posted by Wido den Hollander <wi...@widodh.nl>.
On 01/07/2016 03:47 PM, Nux! wrote:
>> And that is indeed it. I'm not keen on doing point releases either.
>
>> Simplify upgrade paths, ditch the fact that a VR has to be upgraded
>> every time, etc, etc.
>
> That'd be heaven. I hate the VR upgrades!
>
A version might have a flag telling if the VR needs upgrade or not.
Boolean, true/false.
>>
>> We could also call 4.8 simply 4.7.2. It's just a number :)
>
> +1, perhaps switch to version based on dates?
>
No, my point is. Just the fact that it is called 4.8 makes people itchy.
I really wonder why.
We could also call it 5.0 and that will keep people away from it?
Wido
>>
>> Wido
>
>
Re: Minor releases!
Posted by Nux! <nu...@li.nux.ro>.
> And that is indeed it. I'm not keen on doing point releases either.
> Simplify upgrade paths, ditch the fact that a VR has to be upgraded
> every time, etc, etc.
That'd be heaven. I hate the VR upgrades!
>
> We could also call 4.8 simply 4.7.2. It's just a number :)
+1, perhaps switch to version based on dates?
>
> Wido
Re: Minor releases!
Posted by Wido den Hollander <wi...@widodh.nl>.
On 01/07/2016 03:35 PM, Remi Bergsma wrote:
>
> We released:
> 4.6.0
> 4.6.1
> 4.6.2
> 4.7.0
> (4.7.1)
> (4.8.0)
>
> We _do_ release minor releases, and every month one with new features.
>
> To be clear:
> Any branch (be it 4.5, 4.6, 4.7) can have as many minor releases as people want, until the end of time. Just step up and release it.
>
And that is indeed it. I'm not keen on doing point releases either.
Simplify upgrade paths, ditch the fact that a VR has to be upgraded
every time, etc, etc.
We could also call 4.8 simply 4.7.2. It's just a number :)
Wido
> Regards,
>
> Remi
>
>
>
>
>> We are not yet on 4.6, so we have downtime by upgrading the routers, we
>> have approx 120 VR, which take about ~ 5-10 minutes of downtime each.
>>
>> We are not agile in upgrading, we have freeze times and plan our
>> releases and test each release intensively (it saved our asses in the past).
>>
>> Every bigger software project in the world has agreed that providing
>> minor releases is good practice:
>>
>> - New features --> Major release
>> - Fixes --> Minor release
>>
>> The problem with new features are potentially new bugs (by nature):
>>
>> Now you are basically saying:
>>
>> - New features and fixes --> Major release only
>>
>> By definition, you will end up it a more unstable software because you
>> will only get fixes in cost of new features and potentially new bugs. I
>> don't call this stable.
>>
>> I do not see any benefit of the current release rush.
>>
>> René
Re: Minor releases!
Posted by Remi Bergsma <RB...@schubergphilis.com>.
We released:
4.6.0
4.6.1
4.6.2
4.7.0
(4.7.1)
(4.8.0)
We _do_ release minor releases, and every month one with new features.
To be clear:
Any branch (be it 4.5, 4.6, 4.7) can have as many minor releases as people want, until the end of time. Just step up and release it.
Regards,
Remi
>We are not yet on 4.6, so we have downtime by upgrading the routers, we
>have approx 120 VR, which take about ~ 5-10 minutes of downtime each.
>
>We are not agile in upgrading, we have freeze times and plan our
>releases and test each release intensively (it saved our asses in the past).
>
>Every bigger software project in the world has agreed that providing
>minor releases is good practice:
>
>- New features --> Major release
>- Fixes --> Minor release
>
>The problem with new features are potentially new bugs (by nature):
>
>Now you are basically saying:
>
>- New features and fixes --> Major release only
>
>By definition, you will end up it a more unstable software because you
>will only get fixes in cost of new features and potentially new bugs. I
>don't call this stable.
>
>I do not see any benefit of the current release rush.
>
>René
Re: Minor releases!
Posted by Rene Moser <ma...@renemoser.net>.
Hi Remi
On 01/07/2016 02:28 PM, Remi Bergsma wrote:
> I simply don’t understand why you need lots and lots of minor versions. I do understand you need a stable cloud, and that’s exactly what we’re achieving here.
>
> We changed our way of working from 4.6 on. Before that, it took _a long_ time to release new versions (be it major, minor or patch). Releases were not high quality so people waited for .1 or .2 to be released. When you did eventually upgrade, you’d hit major trouble. It was simply not OK. And many minor releases don’t help here. The root cause of the trouble is that the whole idea of branching off a new version and have a QA team make it stable (while the rest continues on master) doesn’t make sense (any more). That’s why in the summer of 2015 we decided to stabilise master instead and release from that. [1] One final time it took a great effort to make a branch stable.
>
> We released 4.6 in Nov 2015
> We released 4.7 in Dec 2015
> We will release 4.8 in Jan 2016 (unless people think I should stop doing this)
>
> In between, we submitted patch releases to quickly address bugs (4.6.1, 4.6.2 etc).
What about 4.5? 4.4? Is 4.6 still getting 4.6.3? Users don't see that.
> You know what? It’s easy to do these upgrades. Much much easier than before. The change is small, the procedure is quick.
We are not yet on 4.6, so we have downtime by upgrading the routers, we
have approx 120 VR, which take about ~ 5-10 minutes of downtime each.
We are not agile in upgrading, we have freeze times and plan our
releases and test each release intensively (it saved our asses in the past).
Every bigger software project in the world has agreed that providing
minor releases is good practice:
- New features --> Major release
- Fixes --> Minor release
The problem with new features are potentially new bugs (by nature):
Now you are basically saying:
- New features and fixes --> Major release only
By definition, you will end up it a more unstable software because you
will only get fixes in cost of new features and potentially new bugs. I
don't call this stable.
I do not see any benefit of the current release rush.
René
Re: Minor releases!
Posted by Rene Moser <ma...@renemoser.net>.
On 01/07/2016 05:04 PM, sebgoa wrote:
> Yet folks (like Rene) may like a pattern of just minor and very infrequent major. While folks like Remi want continuous deployment.
>
> So at the cost of sounding a bit "fatherly" we indeed need to discuss this a bit. I mentioned it after 4.6, and communicate clearly to everyone how this works.
>
> Keeping in mind that we don't want to abandon anyone and we want everybody to be able to upgrade easily.
Hmm no it is not quite right.
I am not against frequent majors, but I my use case needs releases to be
maintained longer then the next major release.
So I think the best solution would be every 12 months a LTS releases
maintained for 18 months for users having such conditions.
I'll get in touch with Rohit.
Regards
René
Re: Minor releases!
Posted by sebgoa <ru...@gmail.com>.
see inline
On Jan 7, 2016, at 3:39 PM, Nux! <nu...@li.nux.ro> wrote:
> Hi,
>
>
>> So, yes, monthly releases can be done and the quality is better than before.
>> Actually, I think we should go much faster. Whenever a PR is merged that fixes
>> your issue, it should be possible to deploy it right away. It’s a change in
>> mindset.
>
>>
>>
>> Let me know!
>
> Ok, so first of all I am letting you know that I appreciate your efforts very much and can't thank you enough for your being RM.
> Also a lot of thanks to other SBP staff, PCExtreme, Leaseweb & Shapeblue and others for all their work; sometimes I just feel like a scavenger, "profiting" from their work. :)
> Keep up the good job!
>
+1000
CloudStack is in its best shape ever indeed due to the hard work from Remi, Wilder, Daan, Rajani, Wido and a few others that I am missing.
It is also in its best shape ever because we moved from developing on master and GA'ing a release branch over several months.
We know have a stable master that also has the latest features and fixes.
> Secondly, my own problem with the current velocity is that it does generate fatigue.
> Everything must be tested all the time, eyes must be on git and on the mailing list.
>
The way I see it we completed the first phase of a big change for us. We changed our release process.
This has led us to this amazing place:
Today we can release production ready CloudStack *at any time* in 72 hours (the time to close a vote), it's terrific.
The other side of the coin is that we can release so often we need to change our mindset and doing so be mindful of folks that are used to the old release pattern.
Clearly we cannot keep maintaining all release branches, we used to say that we would maintain the last two major and the last minor.
Yet folks (like Rene) may like a pattern of just minor and very infrequent major. While folks like Remi want continuous deployment.
So at the cost of sounding a bit "fatherly" we indeed need to discuss this a bit. I mentioned it after 4.6, and communicate clearly to everyone how this works.
Keeping in mind that we don't want to abandon anyone and we want everybody to be able to upgrade easily.
> And this is how it should be, really, but it can become a problem or at least disappointing when people fail to engage as much as we'd want.
> We need to be aware the community is indeed small, the users might be many but few get involved in development (people still don't get "open source") and not many of us have the luxury of
> working with Cloudstack every day and have it be the main subject.
> There are days when I don't even get to think about it, let alone do anything about it, I work some very busy shifts dealing with random things.
>
> I think people in my situation - or worse - might feel uncomfortable with a fast pace, they might feel left behind and sometimes it's just not easy to keep up.
> I know how heavy it feels to run a mission critical cloud and have no idea whether you'll be able to pull the next upgrade off.
> The idea of longer term support through many minor versions can sound very appealing and make people feel cosy. RedHat is exploiting exactly this with their RHEL, for example.
>
> Having said that, I would not go back to the old ways. I think we're on a good track here and we just need to test more, especially automate said testing more, in environments as close to production as we can. This my 2016 resolution. :-)
+1000 here.
We are on a great track. Let's keep refining our process, complete the second phase which is proper CI , agree on how to move forward and communicate on which releases are maintained and for how long.
>
> My 2 pence
>
> Lucian
Re: Minor releases!
Posted by Nux! <nu...@li.nux.ro>.
Hi,
> So, yes, monthly releases can be done and the quality is better than before.
> Actually, I think we should go much faster. Whenever a PR is merged that fixes
> your issue, it should be possible to deploy it right away. It’s a change in
> mindset.
This actually sounds very interesting; having a major.minor.$date or something, but it could get trickier when the said PR fixes something in the DB and the versions need to be gracefully dealt with.
>
>
> Before this change, master was broken all the time. Now you can even run
> production from master. It just works (tm). Users should pick the latest
> version.
I've watched ACS mature over the years and now it's definitely in its best shape yet.
> There will always be bugs, so the faster we release, the smaller the
> diff and the easier the deploy and best of all: the sooner the bug is resolved
> in production.
Makes sense.
>
> To be honest, I have the feeling not much people agree with me. So maybe I
> should stop doing this. I also see it in the number of people reviewing and
> testing. It’s simply disappointing. Anyway, let me know what you guys want. If
> people want to go back to the old way of working then that’s also fine with me.
> I just won’t be the Release Manager of it.
>
> Let me know!
Ok, so first of all I am letting you know that I appreciate your efforts very much and can't thank you enough for your being RM.
Also a lot of thanks to other SBP staff, PCExtreme, Leaseweb & Shapeblue and others for all their work; sometimes I just feel like a scavenger, "profiting" from their work. :)
Keep up the good job!
Secondly, my own problem with the current velocity is that it does generate fatigue.
Everything must be tested all the time, eyes must be on git and on the mailing list.
And this is how it should be, really, but it can become a problem or at least disappointing when people fail to engage as much as we'd want.
We need to be aware the community is indeed small, the users might be many but few get involved in development (people still don't get "open source") and not many of us have the luxury of
working with Cloudstack every day and have it be the main subject.
There are days when I don't even get to think about it, let alone do anything about it, I work some very busy shifts dealing with random things.
I think people in my situation - or worse - might feel uncomfortable with a fast pace, they might feel left behind and sometimes it's just not easy to keep up.
I know how heavy it feels to run a mission critical cloud and have no idea whether you'll be able to pull the next upgrade off.
The idea of longer term support through many minor versions can sound very appealing and make people feel cosy. RedHat is exploiting exactly this with their RHEL, for example.
Having said that, I would not go back to the old ways. I think we're on a good track here and we just need to test more, especially automate said testing more, in environments as close to production as we can. This my 2016 resolution. :-)
My 2 pence
Lucian
Re: Minor releases!
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi René, all,
I simply don’t understand why you need lots and lots of minor versions. I do understand you need a stable cloud, and that’s exactly what we’re achieving here.
We changed our way of working from 4.6 on. Before that, it took _a long_ time to release new versions (be it major, minor or patch). Releases were not high quality so people waited for .1 or .2 to be released. When you did eventually upgrade, you’d hit major trouble. It was simply not OK. And many minor releases don’t help here. The root cause of the trouble is that the whole idea of branching off a new version and have a QA team make it stable (while the rest continues on master) doesn’t make sense (any more). That’s why in the summer of 2015 we decided to stabilise master instead and release from that. [1] One final time it took a great effort to make a branch stable.
We released 4.6 in Nov 2015
We released 4.7 in Dec 2015
We will release 4.8 in Jan 2016 (unless people think I should stop doing this)
In between, we submitted patch releases to quickly address bugs (4.6.1, 4.6.2 etc).
You know what? It’s easy to do these upgrades. Much much easier than before. The change is small, the procedure is quick. I upgraded our production environment from 4.6.2 to 4.7.0 in under 10 minutes (a clustered setup). When 4.7.0 got released, for the first time ever, we knew 100% sure it included everything from 4.6 as it was built on top of it. No regression between 4.6 and 4.7. Actually there’s no need to use 4.6 any more, when 4.7 is out. It’s essentially the same, plus new features.
So, yes, monthly releases can be done and the quality is better than before. Actually, I think we should go much faster. Whenever a PR is merged that fixes your issue, it should be possible to deploy it right away. It’s a change in mindset.
Before this change, master was broken all the time. Now you can even run production from master. It just works (tm). Users should pick the latest version.
If someone wants to maintain an older release, that can be done. I know Rohit maintains older versions for ShapeBlue customers and that makes sense. Until everyone gets used to an agile way of working this may still be needed. I just don’t work that way any more.
To summarise, you want a stable cloud and I think since 4.6 we provide just that. Quick releases, fast delivery of new features and even faster delivery of bug fixes. There will always be bugs, so the faster we release, the smaller the diff and the easier the deploy and best of all: the sooner the bug is resolved in production.
To be honest, I have the feeling not much people agree with me. So maybe I should stop doing this. I also see it in the number of people reviewing and testing. It’s simply disappointing. Anyway, let me know what you guys want. If people want to go back to the old way of working then that’s also fine with me. I just won’t be the Release Manager of it.
Let me know!
Regards,
Remi
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Release+principles+for+Apache+CloudStack+4.6+and+up
PS: At work we run multiple CloudStack powered clouds with hundreds of hypervisors so I think I know what I'm talking about.
On 07/01/16 13:39, "Rene Moser" <ma...@renemoser.net> wrote:
>Hi
>
>I don't want to be the Grinch but this new release cycle feels not
>matching users needs. Especially not my employers one.
>
>We are releasing major versions every month? Great! So we release a
>minors every week/day!? No, we don't! :( Every couple of months actually.
>
>A release not getting minors is not a release! It is a dead horse.
>
>Questions:
>
>* Who will use 4.6? when 4.8 was released?
>* Is 4.6 still maintained, is 4.5 maintained? is 4.4 maintained?
>* Do we really think, clouds will be major upgraded every month?
>* If we do a major release every month, why is the last minor 4.5.2 in
>August 2015 and not 4.5.3 or 4.5.4?
>
>And if you think there were no commits since 4.5.2?
>
>git rev-list 4.5.2..HEAD --count
>77
>
>Branch 4.4:
>git rev-list 4.4.4..HEAD --count
>8
>
>At the end users are more confused, which version to take. We are
>dealing with _infrastructure_.
>
>We must rethink the major release cycling, if it impacts minor releases.
>I need a stable cloud aka lots and lots of minors and those now.
>
>Regards
>René
>