[TTH] Why do you need release candidates?

[Bertrand Delacretaz <bd...@apache.org>]

Hi Flex team,

Reviewing your activities to try and help this community run in a
smoother way, I'm wondering why you need release candidates.

Those were useful during your Apache incubation, to allow "total
strangers" who don't have the right tools for example to review your
releases, but in a small team that does have the appropriate tools to
verify things form source I'm not sure they are worth the effort.

Release candidates create a lot of work for the release manager, and
require a lot of coordination between yourselves to find out when to
create another one, cast multiple votes or argue about when it's
appropriate to carry votes over, etc.

Based on other projects, I'm suggesting a simpler way of creating your
releases, below.

As usual, this is only my old fart^H^H^H^H experienced Apache member
advice, feel free to take it into account or not.

You might also just try it on one of your releases to see if it works
- it's easy to try.

-Bertrand



Here's my suggested release (non-) process:

Designate a Git branch to become the release and create a jira version
number that represents it.

Create small, specific jira issues for things (including integrating
patches from other branches) that prevent the branch from being
released as is, and link those to the release version in jira, so that
simple jira queries show what’s left to do and what's been done.

Setup your continuous integration system to run regularly on that
branch, ideally after every commit.

Ask people to review the branch and create jira issues for anything
wrong with it.

Wait until all jira issues are closed (or rescheduled), prepare the
release artifacts, vote on them and release.

You vote just once, you’re not arguing all the time about where the
release stands (or if you do that’s in or about specific jira issue
which make things clearer), and the release vote passes 99% of the
time.

Re: [TTH] Why do you need release candidates?

[Erik de Bruin <er...@ixsoftware.nl>]

We don't mention it explicitly in the 'less-RC' article (since it's a
long time and good practice Justin set up way back when), but we also
use your suggested method of using JIRA tickets to track the release
progress [1].

EdB

1: https://issues.apache.org/jira/browse/FLEX-34560



On Sat, Dec 6, 2014 at 10:33 AM, Bertrand Delacretaz
<bd...@apache.org> wrote:
> On Sat, Dec 6, 2014 at 10:13 AM, Erik de Bruin <er...@ixsoftware.nl> wrote:
>> ...that is a very accurate summary of the 'less-RC' process [1]
>> we've been working on for our future releases...
>
> Oh cool! For some reason that felt more complicated when I looked at
> it - I'll have another look after the week-end!
>
> -Bertrand
>
>
>> 1: https://cwiki.apache.org/confluence/x/2oH0Ag



-- 
Ix Multimedia Software

Jan Luykenstraat 27
3521 VB Utrecht

T. 06-51952295
I. www.ixsoftware.nl

Re: [TTH] Why do you need release candidates?

[Bertrand Delacretaz <bd...@apache.org>]

On Sat, Dec 6, 2014 at 10:13 AM, Erik de Bruin <er...@ixsoftware.nl> wrote:
> ...that is a very accurate summary of the 'less-RC' process [1]
> we've been working on for our future releases...

Oh cool! For some reason that felt more complicated when I looked at
it - I'll have another look after the week-end!

-Bertrand


> 1: https://cwiki.apache.org/confluence/x/2oH0Ag

Re: [TTH] Why do you need release candidates?

[Erik de Bruin <er...@ixsoftware.nl>]

Bertrand, that is a very accurate summary of the 'less-RC' process [1]
we've been working on for our future releases.

Thanks,

EdB

1: https://cwiki.apache.org/confluence/x/2oH0Ag



On Sat, Dec 6, 2014 at 10:05 AM, Bertrand Delacretaz
<bd...@apache.org> wrote:
> Hi Flex team,
>
> Reviewing your activities to try and help this community run in a
> smoother way, I'm wondering why you need release candidates.
>
> Those were useful during your Apache incubation, to allow "total
> strangers" who don't have the right tools for example to review your
> releases, but in a small team that does have the appropriate tools to
> verify things form source I'm not sure they are worth the effort.
>
> Release candidates create a lot of work for the release manager, and
> require a lot of coordination between yourselves to find out when to
> create another one, cast multiple votes or argue about when it's
> appropriate to carry votes over, etc.
>
> Based on other projects, I'm suggesting a simpler way of creating your
> releases, below.
>
> As usual, this is only my old fart^H^H^H^H experienced Apache member
> advice, feel free to take it into account or not.
>
> You might also just try it on one of your releases to see if it works
> - it's easy to try.
>
> -Bertrand
>
>
>
> Here's my suggested release (non-) process:
>
> Designate a Git branch to become the release and create a jira version
> number that represents it.
>
> Create small, specific jira issues for things (including integrating
> patches from other branches) that prevent the branch from being
> released as is, and link those to the release version in jira, so that
> simple jira queries show what’s left to do and what's been done.
>
> Setup your continuous integration system to run regularly on that
> branch, ideally after every commit.
>
> Ask people to review the branch and create jira issues for anything
> wrong with it.
>
> Wait until all jira issues are closed (or rescheduled), prepare the
> release artifacts, vote on them and release.
>
> You vote just once, you’re not arguing all the time about where the
> release stands (or if you do that’s in or about specific jira issue
> which make things clearer), and the release vote passes 99% of the
> time.



-- 
Ix Multimedia Software

Jan Luykenstraat 27
3521 VB Utrecht

T. 06-51952295
I. www.ixsoftware.nl

Re: [TTH] Why do you need release candidates?

[Alex Harui <ah...@adobe.com>]

Hi everyone,

At this point, I would like to suggest we table this debate until Bertrand
gets a chance to look at the latest version of the new release process
proposal.  If he’s ok with the proposal, then we put it up for a vote.

Thanks,
-Alex

On 12/7/14, 12:34 AM, "OmPrakash Muppirala" <bi...@gmail.com> wrote:

>On Dec 7, 2014 12:24 AM, "Justin Mclean" <ju...@classsoftware.com> wrote:
>>
>> Hi,
>>
>> > 'Fairly' accurate is not good enough.
>>
>> I not done the stats on every release but you would have to agree it's
>happened many times. The fact we don't have  only 1 or 2 RC for each
>release confirms it. I'd estimate that 80%+ of the releases  have had more
>than 2 RCs I think "fairly accurate" covers that.
>
>I meant your statement about the PMC, not the stats.
>
>>
>> > The point is, by painting with such broad strokes, you are discounting
>the
>> > efforts of those who take time to contribute to the project.
>>
>> I'm not discounting anyones effort there and I really not know why you
>would assume or state that.
>
>That is what your language suggests.  Which is what many of us have been
>trying to tell you.
>
>>
>> We do have some issue here. For instance:
>> - How many people have added new mustella tests?
>> - How many new mustella tests have been added since incubation?
>> - How many PMC members run the tests themselves on the RC?
>> - How many committers run the mustella test before checking anything in?
>> - How many committers/PMC members even know how to run the tests?
>> - We have regular failures of the mustella test runs - often for no
>explained reasons
>> - We need to rerun failed tests several times just to get all the tests
>to pass - that's hardly ideal
>> - We've run into mustella tests that are wrong and just working by
>accident
>
>None of what you mentioned are problems with  Mustella itself.
>
>>
>> Yes the CI solves some of the later points and I totally agree having
>tests (of any sort) is better than not having them but it's fair to
>describe it as an imperfect solution.
>
>That is not what you said.  You blamed the release process on Mustella.
>
>>
>> > But there is no point in blaming the CI system or the Mustella tests
>>for
>> > our difficult release process.
>>
>> It has caused issues in the past ie Michael + Nick collection changes.
>
>Because the changes broke existing tests.  What would be different if
>those
>were FlexUnit or JUnit tests?
>
>Thanks,
>Om
>
>>
>> Thanks,
>> Justin

Re: [TTH] Why do you need release candidates?

[OmPrakash Muppirala <bi...@gmail.com>]

On Dec 7, 2014 12:24 AM, "Justin Mclean" <ju...@classsoftware.com> wrote:
>
> Hi,
>
> > 'Fairly' accurate is not good enough.
>
> I not done the stats on every release but you would have to agree it's
happened many times. The fact we don't have  only 1 or 2 RC for each
release confirms it. I'd estimate that 80%+ of the releases  have had more
than 2 RCs I think "fairly accurate" covers that.

I meant your statement about the PMC, not the stats.

>
> > The point is, by painting with such broad strokes, you are discounting
the
> > efforts of those who take time to contribute to the project.
>
> I'm not discounting anyones effort there and I really not know why you
would assume or state that.

That is what your language suggests.  Which is what many of us have been
trying to tell you.

>
> We do have some issue here. For instance:
> - How many people have added new mustella tests?
> - How many new mustella tests have been added since incubation?
> - How many PMC members run the tests themselves on the RC?
> - How many committers run the mustella test before checking anything in?
> - How many committers/PMC members even know how to run the tests?
> - We have regular failures of the mustella test runs - often for no
explained reasons
> - We need to rerun failed tests several times just to get all the tests
to pass - that's hardly ideal
> - We've run into mustella tests that are wrong and just working by
accident

None of what you mentioned are problems with  Mustella itself.

>
> Yes the CI solves some of the later points and I totally agree having
tests (of any sort) is better than not having them but it's fair to
describe it as an imperfect solution.

That is not what you said.  You blamed the release process on Mustella.

>
> > But there is no point in blaming the CI system or the Mustella tests for
> > our difficult release process.
>
> It has caused issues in the past ie Michael + Nick collection changes.

Because the changes broke existing tests.  What would be different if those
were FlexUnit or JUnit tests?

Thanks,
Om

>
> Thanks,
> Justin

Re: [TTH] Why do you need release candidates?

[Justin Mclean <ju...@classsoftware.com>]

Hi,

> 'Fairly' accurate is not good enough.

I not done the stats on every release but you would have to agree it's happened many times. The fact we don't have  only 1 or 2 RC for each release confirms it. I'd estimate that 80%+ of the releases  have had more than 2 RCs I think "fairly accurate" covers that.

> The point is, by painting with such broad strokes, you are discounting the
> efforts of those who take time to contribute to the project.

I'm not discounting anyones effort there and I really not know why you would assume or state that.

We do have some issue here. For instance:
- How many people have added new mustella tests?
- How many new mustella tests have been added since incubation?
- How many PMC members run the tests themselves on the RC?
- How many committers run the mustella test before checking anything in?
- How many committers/PMC members even know how to run the tests?
- We have regular failures of the mustella test runs - often for no explained reasons
- We need to rerun failed tests several times just to get all the tests to pass - that's hardly ideal
- We've run into mustella tests that are wrong and just working by accident

Yes the CI solves some of the later points and I totally agree having tests (of any sort) is better than not having them but it's fair to describe it as an imperfect solution.

> But there is no point in blaming the CI system or the Mustella tests for
> our difficult release process.

It has caused issues in the past ie Michael + Nick collection changes.

Thanks,
Justin

Re: [TTH] Why do you need release candidates?

[OmPrakash Muppirala <bi...@gmail.com>]

On Dec 6, 2014 9:13 PM, "Justin Mclean" <ju...@classsoftware.com> wrote:
>
> Hi,
>
> > You are doing it again :-)
>
> IMO It was a fairly accurate representation of the state of affairs. You
do realise when I write "PMC" I'm including myself? For instance look at
the amount of feedback from the the "test" RC of TourDeFlex (RC0) and then
the issues found in RC1 and RC2. Other releases have has similar issues, ie
that important issues or regressions are only found a few RCs in. Our CI
doesn't catch all issues. The RC0 step was added to try and get people to
try stuff out before the first proper RC and cut down on the number of RC
cycles - and IMO it's had some positive effect but not a huge amount.

'Fairly' accurate is not good enough.

The point is, by painting with such broad strokes, you are discounting the
efforts of those who take time to contribute to the project.  I request you
to avoid these kind of blanket statements.

>
> > What are the alternatives to CI and Mustella tests?
>
> There no real good alternative to CI as manual testing take up a lot of
time. With the installer and several other projects don't have tests that
the CI can run *, so someone would need to step forward and write tests for
those project. (Perhaps that not seen as important enough?) This has
happened with TLF (which was donated with no tests) which is great. On that
subject (and while it is manual testing) TourDeFlex is a good way of
testing a new SDK, particularly in a few areas where we have limited or no
tests eg OSMF or newer Apache SDK features.
>
> Having a set of FlexUnit tests rather than Mustella tests would probably
get more people involved in helping out with tests, and would make testing
easier, but that would be a (very) large undertaking. Having a set of tests
that could be run more easily and that didn't take 8 hours would also help,
a few people has suggested some solutions (ie run tests in parallel) but
also hard to implement.
>

Exactly.  We must be happy that we have at least the Mustella tests to keep
the regression bugs away.  Anything more than that is definitely welcome.
But there is no point in blaming the CI system or the Mustella tests for
our difficult release process.

Thanks,
Om

> Thanks,
> Justin
>
> * Ant for AIR used in the installer does have tests but the installer
itself doesn't

Re: [TTH] Why do you need release candidates?

[Justin Mclean <ju...@classsoftware.com>]

Hi,

> You are doing it again :-)

IMO It was a fairly accurate representation of the state of affairs. You do realise when I write "PMC" I'm including myself? For instance look at the amount of feedback from the the "test" RC of TourDeFlex (RC0) and then the issues found in RC1 and RC2. Other releases have has similar issues, ie that important issues or regressions are only found a few RCs in. Our CI doesn't catch all issues. The RC0 step was added to try and get people to try stuff out before the first proper RC and cut down on the number of RC cycles - and IMO it's had some positive effect but not a huge amount.

> What are the alternatives to CI and Mustella tests?

There no real good alternative to CI as manual testing take up a lot of time. With the installer and several other projects don't have tests that the CI can run *, so someone would need to step forward and write tests for those project. (Perhaps that not seen as important enough?) This has happened with TLF (which was donated with no tests) which is great. On that subject (and while it is manual testing) TourDeFlex is a good way of testing a new SDK, particularly in a few areas where we have limited or no tests eg OSMF or newer Apache SDK features.

Having a set of FlexUnit tests rather than Mustella tests would probably get more people involved in helping out with tests, and would make testing easier, but that would be a (very) large undertaking. Having a set of tests that could be run more easily and that didn't take 8 hours would also help, a few people has suggested some solutions (ie run tests in parallel) but also hard to implement.

Thanks,
Justin

* Ant for AIR used in the installer does have tests but the installer itself doesn't

Re: [TTH] Why do you need release candidates?

[OmPrakash Muppirala <bi...@gmail.com>]

On Dec 6, 2014 4:08 PM, "Justin Mclean" <ju...@classsoftware.com> wrote:
>
> Hi,
>
> > Here's my suggested release (non-) process.
>
> As far as I can see we been doing all of that with the old process [1]
with the exception that we usually make multiple RCs. Here's an example
JIRA for the 4.12 release. [2]
>
> So why multiple RCs? The PMC usually

You are doing it again :-)

puts off testing / checking things out until a vote is called, and few
users that use the nightlys or provide feedback on them. So only having one
RC is usually not  possible as issue are only found during the RC process.
We've has a single RC once I think (the Squiggly 1.0 release), but the
installer always seems to give us trouble (most likely due to the large
number of platforms and combinations that require manual testing).

Right, which is why we want to try this new process.

>
> Re the SDK, the CI is complicated in that the tests can be fragile and
take 8 hours to run. Sometime it's not obvious when the tests do fail (and
that's often) if it's code changes, issues with the tests or issue with the
build machine. The Mustella can also sometime difficult to debug and only a
couple of people have experience with that.

What are the alternatives to CI and Mustella tests?

Thanks,
Om

Re: [TTH] Why do you need release candidates?

[Justin Mclean <ju...@classsoftware.com>]

Hi,

> Here's my suggested release (non-) process.

As far as I can see we been doing all of that with the old process [1] with the exception that we usually make multiple RCs. Here's an example JIRA for the 4.12 release. [2]

So why multiple RCs? The PMC usually puts off testing / checking things out until a vote is called, and few users that use the nightlys or provide feedback on them. So only having one RC is usually not  possible as issue are only found during the RC process. We've has a single RC once I think (the Squiggly 1.0 release), but the installer always seems to give us trouble (most likely due to the large number of platforms and combinations that require manual testing).

Re the SDK, the CI is complicated in that the tests can be fragile and take 8 hours to run. Sometime it's not obvious when the tests do fail (and that's often) if it's code changes, issues with the tests or issue with the build machine. The Mustella can also sometime difficult to debug and only a couple of people have experience with that.

Thanks,
Justin

1. https://cwiki.apache.org/confluence/display/FLEX/Release+Guide+for+the+SDK
2. https://issues.apache.org/jira/browse/FLEX-33950