On Apache Mesos release process

[Alex Rukletsov <al...@mesosphere.com>]

Folks,

for more than a year Apache Mesos releases are done according to our "then
new" release policy [1]. It seems to work quite well, but today I would
like to address things that can be improved.

Let's start with pain points:
* A minor bug can cancel a release vote, even for a patch release.
* More canceled votes lead to more RCs and hence create more work for
committers and voters.
* Demotivation for release on a candidate unless other people vote.
* Releases often run behind schedule.

I would like to suggest some improvements to the process:

1. Stricter time releases. The next release should go into planning (with
release managers elected) right after the current is cut. Feature owners
work with the release managers prior to the cut to track progress (k8s
community aims for 2-3 meeting per week discussing blockers and schedule).
This way release managers should have a satisfactory understanding which
new features are going in and what can slow down the release several days
before the cut.

2. Written guideline for which issues can '-1' the release. Though it is up
to the voter how to vote, a clear guideline will set reasonable
expectations and hopefully help us decrease the number of RCs. Regressions
(security, performance, compatibility, functional) can cause -1.
Regressions of experimental features cannot cause -1. Patch releases can be
-1'd in exceptional cases, e.g., critical bug fix missing in the last patch
release. New features cannot block a release.

Note: We love reasonable -1 votes! It is so much better to defer a release
than discover a critical regression from a production user report!

3. Release managers decides what is back ported to the RC branch once it is
cut (same for patch releases). Feature owners and committers are encouraged
to update the release managers timely on the status and importance of
features and bug fixes.

And of course, I encourage everyone using Mesos to test & vote on release
candidates! Identical cluster configurations are rare, each new setup helps
with finding bugs and hence build better software.

[1] https://github.com/apache/mesos/blob/master/docs/versioning.md

Alex.

Re: On Apache Mesos release process

[Vinod Kone <vi...@apache.org>]

Thanks for starting the discussion around on this Alex! Much appreciated
and needed.

I agree with all the points here :) I'm a big proponent of predictable time
based releases.

As an aside, should we spin up a working group for releases? Given the
frequency of our releases and burn down meetings needed, I think it will be
an active and vibrant group. In addition to the tactical aspects of
releases, this group can also come up with guidelines for releases,
improvements etc.

Thoughts?

On Sat, Jun 17, 2017 at 10:25 AM, Alex Rukletsov <al...@mesosphere.com>
wrote:

> Folks,
>
> for more than a year Apache Mesos releases are done according to our "then
> new" release policy [1]. It seems to work quite well, but today I would
> like to address things that can be improved.
>
> Let's start with pain points:
> * A minor bug can cancel a release vote, even for a patch release.
> * More canceled votes lead to more RCs and hence create more work for
> committers and voters.
> * Demotivation for release on a candidate unless other people vote.
> * Releases often run behind schedule.
>
> I would like to suggest some improvements to the process:
>
> 1. Stricter time releases. The next release should go into planning (with
> release managers elected) right after the current is cut. Feature owners
> work with the release managers prior to the cut to track progress (k8s
> community aims for 2-3 meeting per week discussing blockers and schedule).
> This way release managers should have a satisfactory understanding which
> new features are going in and what can slow down the release several days
> before the cut.
>
> 2. Written guideline for which issues can '-1' the release. Though it is
> up to the voter how to vote, a clear guideline will set reasonable
> expectations and hopefully help us decrease the number of RCs. Regressions
> (security, performance, compatibility, functional) can cause -1.
> Regressions of experimental features cannot cause -1. Patch releases can be
> -1'd in exceptional cases, e.g., critical bug fix missing in the last patch
> release. New features cannot block a release.
>
> Note: We love reasonable -1 votes! It is so much better to defer a release
> than discover a critical regression from a production user report!
>
> 3. Release managers decides what is back ported to the RC branch once it
> is cut (same for patch releases). Feature owners and committers are
> encouraged to update the release managers timely on the status and
> importance of features and bug fixes.
>
> And of course, I encourage everyone using Mesos to test & vote on release
> candidates! Identical cluster configurations are rare, each new setup helps
> with finding bugs and hence build better software.
>
> [1] https://github.com/apache/mesos/blob/master/docs/versioning.md
>
> Alex.
>

Re: On Apache Mesos release process

[Vinod Kone <vi...@apache.org>]

Thanks for starting the discussion around on this Alex! Much appreciated
and needed.

I agree with all the points here :) I'm a big proponent of predictable time
based releases.

As an aside, should we spin up a working group for releases? Given the
frequency of our releases and burn down meetings needed, I think it will be
an active and vibrant group. In addition to the tactical aspects of
releases, this group can also come up with guidelines for releases,
improvements etc.

Thoughts?

On Sat, Jun 17, 2017 at 10:25 AM, Alex Rukletsov <al...@mesosphere.com>
wrote:

> Folks,
>
> for more than a year Apache Mesos releases are done according to our "then
> new" release policy [1]. It seems to work quite well, but today I would
> like to address things that can be improved.
>
> Let's start with pain points:
> * A minor bug can cancel a release vote, even for a patch release.
> * More canceled votes lead to more RCs and hence create more work for
> committers and voters.
> * Demotivation for release on a candidate unless other people vote.
> * Releases often run behind schedule.
>
> I would like to suggest some improvements to the process:
>
> 1. Stricter time releases. The next release should go into planning (with
> release managers elected) right after the current is cut. Feature owners
> work with the release managers prior to the cut to track progress (k8s
> community aims for 2-3 meeting per week discussing blockers and schedule).
> This way release managers should have a satisfactory understanding which
> new features are going in and what can slow down the release several days
> before the cut.
>
> 2. Written guideline for which issues can '-1' the release. Though it is
> up to the voter how to vote, a clear guideline will set reasonable
> expectations and hopefully help us decrease the number of RCs. Regressions
> (security, performance, compatibility, functional) can cause -1.
> Regressions of experimental features cannot cause -1. Patch releases can be
> -1'd in exceptional cases, e.g., critical bug fix missing in the last patch
> release. New features cannot block a release.
>
> Note: We love reasonable -1 votes! It is so much better to defer a release
> than discover a critical regression from a production user report!
>
> 3. Release managers decides what is back ported to the RC branch once it
> is cut (same for patch releases). Feature owners and committers are
> encouraged to update the release managers timely on the status and
> importance of features and bug fixes.
>
> And of course, I encourage everyone using Mesos to test & vote on release
> candidates! Identical cluster configurations are rare, each new setup helps
> with finding bugs and hence build better software.
>
> [1] https://github.com/apache/mesos/blob/master/docs/versioning.md
>
> Alex.
>