How can we support a faster release cadence?

[Stephen Connolly <st...@gmail.com>]

One of the projects I am involved with is the Jenkins project. At Jenkins
we cut a release of Jenkins every wednesday... assuming the test all pass...
Not every release is as stable as people might desire (the tests don't
catch everything), hence there is the LTS release line, but none the less,
there is a major release every 7 days... and if you look at the usage stats
(e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg)
most users actually stick fairly close to the latest release.

I have found that this 7 day release cadence can be really helpful for some
code bases.

When I started to think about could we follow this model for the Maven
project as we move towards Maven 4.0, there is one thing that gets in the
way... namely release votes.

The standard answer is that we could publish snapshots... but those are not
indented for use by users... and where the cadence can help is that these
things can be picked up by users.

So what is it that gets in the way with release votes:

* The 72h "soft" requirement for vote duration

* The actions that a PMC member is required to perform before they can
vote. See http://www.apache.org/dev/release which states:

    > Before voting +1 PMC members are required to download the signed
source code package, compile it as provided, and test the resulting
executable on their own platform, along with also verifying that the
package meets the requirements of the ASF policy on releases.

So how exactly do these things get in the way?

Well as I see it the 72h vote duration isn't necessarily a big deal... we
need some duration of notice about what is going into the release, there
will always be people who feel the duration is either too short or two
long... but with a 7 day cadence and maybe a few hours before the release
manager closes out the vote and then you wait for the release to finished
syncing to the mirrors and then the release manager gets a chance to verify
that the release has synced to at least one mirror... you could easily lose
half a day's duration in that process... oh look the release is out 3.5
days after it was cut... and we're cutting another one in 3.5 days... it is
likely we will not get much meaningful feedback from users in the remaining
3.5 days... so essentially you end up with a ping-pong of break... skip...
fix since if a bleeding edge user finds an issue in 4.0.56 we will have cut
4.0.57 by the time they report it to us and the fix ends up in 4.0.58...
with a shorter vote duration, say 12h, the bleeding edge user reports the
issue, we fix and the next release is the one they can use.

In the context of a fast cadence, where every committer in the community
knows there will be a release on wednesday cut from the last revision that
passed all the tests on the CI system unless there have been no commits
since the last release that meet that criteria, do we need to wait the full
72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast
during those 12h... and if not, well just extend until enough votes are
cast)

I think this is different use case from my understanding of the concerns
that drove the 72h vote duration convention, as this would not be 3 PMC
members who all work for the same company and are in the same location
conspiring to drive their changes into the release... everything would be
happening in the open and a 12h window mid-week should allow at least 4h of
waking time in any TZ.

So the second issue is what a PMC member is required to do before voting...

As a PMC member you are required to

1. Download the source code package
2. Compile it as provided
3. Test the resulting executable on your own platform
4. Verify that the package meets the requirements of the ASF policy on
releases

Do we really have to personally do all that *by hand*?

Why can we not have a trusted build server hosted on Apache hardware do the
download of the source package, compile it as provided and run the
automated acceptance tests (on a range of platforms), the RAT tooling and
perhaps verify that the source code package matches what is in source
control? The trusted build server could then report the results to the
project mailing list and then the PMC members just need to confirm the
build server said OK, review the commits between the last time they looked
at the commits and the tag (which they know matches what is in the source
bundle) and then vote +1?

The PMC members are supposed to be keeping an eye on the commits anyway, so
that shouldn't be too onerous, and the release manager could even provide a
link to the build server confirmation build in the VOTE email.

I would appreciate any people's thoughts on the above.

-Stephen

P.S.
* Speaking in my personal capacity as a member of the ASF.
* I am not saying that Maven will move to such a model, or even wants to
move to such a model... more that I was thinking about the issues that
might prevent us if we so desired... I know other projects at Apache are
interested in fast release cadence however, so getting this topic discussed
in the open is no bad thing IMHO

Re: How can we support a faster release cadence?

[Dave Fisher <da...@comcast.net>]

Keeping this in public.

We VOTE and review precisely to minimize and catch the IP mistakes that inevitably happen when people are examining the IP from time to time. The community will care about quality, but minding the IP protects us all.

RAT check, detailed review and 3 +1 where those involved know how to review the release are required.

I would have a hard time using an Apache project in production if it were continually changing. I want a stable version that has no surprises if I choose to build from trunk I can, at my own risk. I want a version (patched or not) that I have confirmed works for me.

What's doubly funny is that the OP compares this situation to Jenkins a CI system. Many projects have nightly builds for precisely the reasons the OP is asking weekly releases - to help the community.

Oh look - https://builds.apache.org/ - it is Jenkins! and here are the latest and greatest (or not so great) from many projects. These can be shared on the ML but not on project websites precisely to keep the use to the engaged developer community.

Regards,
Dave

On Feb 7, 2014, at 9:35 AM, Alex Harui wrote:

> IMO, a more important question is:  Does it fit within the "Apache Way" of "Community over Code" to have a project release on a particular schedule?  Because it feels to me that if you have a release cadence then you are saying "Clock over Community".
> 
> I get that having a schedule helps many in the community manage their time and expectations, but I am concerned that a schedule without wiggle room means that there is less subjectivity to release quality which can be bad.  No matter how long it takes to get from a Release Manager starting the packaging to getting it on the dist server, someone may find a critical bug in that window.  Telling them that they have to wait for the next release is "fair" but may not serve the best interests of the community.
> 
> Then, once you allow subjectivity in a release quality discussion, then I think you have to give 72 hours because folks are often volunteers and may not have the cycles to respond in a shorter time frame, but may still have something very important to say, even if you have your 3 +1 votes already.
> 
> Similarly, there are questions about whether a release manager should be allowed to PGP sign artifacts created on a CI server.
> 
> And regarding the steps required to approve a release, it has been my experience on every project I've worked on or used that the bug base has a "unable to reproduce" option that gets used often enough to make me believe that a few CI/AutomatedTesting servers are not able to fully certify quality.  IOW, things that work fine on your computer or on the CI or testing servers simply may not work on some community member's computer.   And providing a window of time for finding that out is part of the current release policy.
> 
> If you've ever just missed a bus or train, knowing another one is coming soon isn't very satisfying if you're in a hurry.  If the driver sees you running but looks at his watch and drives away, how does that make you feel about that person and the company he works for?
> 
> Thanks,
> -Alex
> 
> From: Stephen Connolly <st...@gmail.com>>
> Reply-To: "board@apache.org<ma...@apache.org>" <bo...@apache.org>>
> Date: Friday, February 7, 2014 3:02 AM
> To: "dev@community.apache.org<ma...@community.apache.org>" <de...@community.apache.org>>
> Subject: How can we support a faster release cadence?
> 
> One of the projects I am involved with is the Jenkins project. At Jenkins we cut a release of Jenkins every wednesday... assuming the test all pass...
> Not every release is as stable as people might desire (the tests don't catch everything), hence there is the LTS release line, but none the less, there is a major release every 7 days... and if you look at the usage stats (e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg) most users actually stick fairly close to the latest release.
> 
> I have found that this 7 day release cadence can be really helpful for some code bases.
> 
> When I started to think about could we follow this model for the Maven project as we move towards Maven 4.0, there is one thing that gets in the way... namely release votes.
> 
> The standard answer is that we could publish snapshots... but those are not indented for use by users... and where the cadence can help is that these things can be picked up by users.
> 
> So what is it that gets in the way with release votes:
> 
> * The 72h "soft" requirement for vote duration
> 
> * The actions that a PMC member is required to perform before they can vote. See http://www.apache.org/dev/release which states:
> 
>> Before voting +1 PMC members are required to download the signed source code package, compile it as provided, and test the resulting executable on their own platform, along with also verifying that the package meets the requirements of the ASF policy on releases.
> 
> So how exactly do these things get in the way?
> 
> Well as I see it the 72h vote duration isn't necessarily a big deal... we need some duration of notice about what is going into the release, there will always be people who feel the duration is either too short or two long... but with a 7 day cadence and maybe a few hours before the release manager closes out the vote and then you wait for the release to finished syncing to the mirrors and then the release manager gets a chance to verify that the release has synced to at least one mirror... you could easily lose half a day's duration in that process... oh look the release is out 3.5 days after it was cut... and we're cutting another one in 3.5 days... it is likely we will not get much meaningful feedback from users in the remaining 3.5 days... so essentially you end up with a ping-pong of break... skip... fix since if a bleeding edge user finds an issue in 4.0.56 we will have cut 4.0.57 by the time they report it to us and the fix ends up in 4.0.58... with a shorter vote duration, say 12h, the bleeding edge user reports the issue, we fix and the next release is the one they can use.
> 
> In the context of a fast cadence, where every committer in the community knows there will be a release on wednesday cut from the last revision that passed all the tests on the CI system unless there have been no commits since the last release that meet that criteria, do we need to wait the full 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast during those 12h... and if not, well just extend until enough votes are cast)
> 
> I think this is different use case from my understanding of the concerns that drove the 72h vote duration convention, as this would not be 3 PMC members who all work for the same company and are in the same location conspiring to drive their changes into the release... everything would be happening in the open and a 12h window mid-week should allow at least 4h of waking time in any TZ.
> 
> So the second issue is what a PMC member is required to do before voting...
> 
> As a PMC member you are required to
> 
> 1. Download the source code package
> 2. Compile it as provided
> 3. Test the resulting executable on your own platform
> 4. Verify that the package meets the requirements of the ASF policy on releases
> 
> Do we really have to personally do all that *by hand*?
> 
> Why can we not have a trusted build server hosted on Apache hardware do the download of the source package, compile it as provided and run the automated acceptance tests (on a range of platforms), the RAT tooling and perhaps verify that the source code package matches what is in source control? The trusted build server could then report the results to the project mailing list and then the PMC members just need to confirm the build server said OK, review the commits between the last time they looked at the commits and the tag (which they know matches what is in the source bundle) and then vote +1?
> 
> The PMC members are supposed to be keeping an eye on the commits anyway, so that shouldn't be too onerous, and the release manager could even provide a link to the build server confirmation build in the VOTE email.
> 
> I would appreciate any people's thoughts on the above.
> 
> -Stephen
> 
> P.S.
> * Speaking in my personal capacity as a member of the ASF.
> * I am not saying that Maven will move to such a model, or even wants to move to such a model... more that I was thinking about the issues that might prevent us if we so desired... I know other projects at Apache are interested in fast release cadence however, so getting this topic discussed in the open is no bad thing IMHO
> 

Re: How can we support a faster release cadence?

[Eric Covener <co...@gmail.com>]

On Fri, Feb 7, 2014 at 12:35 PM, Alex Harui <ah...@adobe.com> wrote:
> IMO, a more important question is:  Does it fit within the "Apache Way" of "Community over Code" to have a project release on a particular schedule?
> Because it feels to me that if you have a release cadence then you are saying "Clock over Community".

This assumes the schedule isn't for the benefit of the community, or
that it's so rhetorically rigid that it might not be in every
instance.

I don't think on its own it is any less compatible with the "Apache
Way" then not having a particular schedule.  Either extreme can be to
the detriment of the community (like many otherwise mundane things)

Re: How can we support a faster release cadence?

[Alex Harui <ah...@adobe.com>]

IMO, a more important question is:  Does it fit within the "Apache Way" of "Community over Code" to have a project release on a particular schedule?  Because it feels to me that if you have a release cadence then you are saying "Clock over Community".

I get that having a schedule helps many in the community manage their time and expectations, but I am concerned that a schedule without wiggle room means that there is less subjectivity to release quality which can be bad.  No matter how long it takes to get from a Release Manager starting the packaging to getting it on the dist server, someone may find a critical bug in that window.  Telling them that they have to wait for the next release is "fair" but may not serve the best interests of the community.

Then, once you allow subjectivity in a release quality discussion, then I think you have to give 72 hours because folks are often volunteers and may not have the cycles to respond in a shorter time frame, but may still have something very important to say, even if you have your 3 +1 votes already.

Similarly, there are questions about whether a release manager should be allowed to PGP sign artifacts created on a CI server.

And regarding the steps required to approve a release, it has been my experience on every project I've worked on or used that the bug base has a "unable to reproduce" option that gets used often enough to make me believe that a few CI/AutomatedTesting servers are not able to fully certify quality.  IOW, things that work fine on your computer or on the CI or testing servers simply may not work on some community member's computer.   And providing a window of time for finding that out is part of the current release policy.

If you've ever just missed a bus or train, knowing another one is coming soon isn't very satisfying if you're in a hurry.  If the driver sees you running but looks at his watch and drives away, how does that make you feel about that person and the company he works for?

Thanks,
-Alex

From: Stephen Connolly <st...@gmail.com>>
Reply-To: "board@apache.org<ma...@apache.org>" <bo...@apache.org>>
Date: Friday, February 7, 2014 3:02 AM
To: "dev@community.apache.org<ma...@community.apache.org>" <de...@community.apache.org>>
Subject: How can we support a faster release cadence?

One of the projects I am involved with is the Jenkins project. At Jenkins we cut a release of Jenkins every wednesday... assuming the test all pass...
Not every release is as stable as people might desire (the tests don't catch everything), hence there is the LTS release line, but none the less, there is a major release every 7 days... and if you look at the usage stats (e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg) most users actually stick fairly close to the latest release.

I have found that this 7 day release cadence can be really helpful for some code bases.

When I started to think about could we follow this model for the Maven project as we move towards Maven 4.0, there is one thing that gets in the way... namely release votes.

The standard answer is that we could publish snapshots... but those are not indented for use by users... and where the cadence can help is that these things can be picked up by users.

So what is it that gets in the way with release votes:

* The 72h "soft" requirement for vote duration

* The actions that a PMC member is required to perform before they can vote. See http://www.apache.org/dev/release which states:

    > Before voting +1 PMC members are required to download the signed source code package, compile it as provided, and test the resulting executable on their own platform, along with also verifying that the package meets the requirements of the ASF policy on releases.

So how exactly do these things get in the way?

Well as I see it the 72h vote duration isn't necessarily a big deal... we need some duration of notice about what is going into the release, there will always be people who feel the duration is either too short or two long... but with a 7 day cadence and maybe a few hours before the release manager closes out the vote and then you wait for the release to finished syncing to the mirrors and then the release manager gets a chance to verify that the release has synced to at least one mirror... you could easily lose half a day's duration in that process... oh look the release is out 3.5 days after it was cut... and we're cutting another one in 3.5 days... it is likely we will not get much meaningful feedback from users in the remaining 3.5 days... so essentially you end up with a ping-pong of break... skip... fix since if a bleeding edge user finds an issue in 4.0.56 we will have cut 4.0.57 by the time they report it to us and the fix ends up in 4.0.58... with a shorter vote duration, say 12h, the bleeding edge user reports the issue, we fix and the next release is the one they can use.

In the context of a fast cadence, where every committer in the community knows there will be a release on wednesday cut from the last revision that passed all the tests on the CI system unless there have been no commits since the last release that meet that criteria, do we need to wait the full 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast during those 12h... and if not, well just extend until enough votes are cast)

I think this is different use case from my understanding of the concerns that drove the 72h vote duration convention, as this would not be 3 PMC members who all work for the same company and are in the same location conspiring to drive their changes into the release... everything would be happening in the open and a 12h window mid-week should allow at least 4h of waking time in any TZ.

So the second issue is what a PMC member is required to do before voting...

As a PMC member you are required to

1. Download the source code package
2. Compile it as provided
3. Test the resulting executable on your own platform
4. Verify that the package meets the requirements of the ASF policy on releases

Do we really have to personally do all that *by hand*?

Why can we not have a trusted build server hosted on Apache hardware do the download of the source package, compile it as provided and run the automated acceptance tests (on a range of platforms), the RAT tooling and perhaps verify that the source code package matches what is in source control? The trusted build server could then report the results to the project mailing list and then the PMC members just need to confirm the build server said OK, review the commits between the last time they looked at the commits and the tag (which they know matches what is in the source bundle) and then vote +1?

The PMC members are supposed to be keeping an eye on the commits anyway, so that shouldn't be too onerous, and the release manager could even provide a link to the build server confirmation build in the VOTE email.

I would appreciate any people's thoughts on the above.

-Stephen

P.S.
* Speaking in my personal capacity as a member of the ASF.
* I am not saying that Maven will move to such a model, or even wants to move to such a model... more that I was thinking about the issues that might prevent us if we so desired... I know other projects at Apache are interested in fast release cadence however, so getting this topic discussed in the open is no bad thing IMHO

Re: How can we support a faster release cadence?

[Marvin Humphrey <ma...@rectangular.com>]

On Sat, Feb 8, 2014 at 8:50 AM, Benson Margulies <bi...@gmail.com> wrote:
> So it's not true to say that 'community over code' precludes this.
> Some communities have chosen this.

+1 high-cadence releasing should be a community decision.

> All the remarks about code quality and stability are, I think, off
> topic here. The reason for the release system is legal, not QA. If
> people think that more releases, or more automated releases, will
> decrease quality, then they shouldn't adopt those patterns in their
> communities.

Proponents of high-cadence releasing generally make the argument that it
increases quality.

    http://martinfowler.com/books/continuousDelivery.html

    Long, high intensity releases become a thing of the past.

    http://www.stephen-smith.co.uk/release-more-with-less/

    However, the benefits of releasing smaller change sets more frequently -
    improved feedback, risk, overheards, efficiency, and ownership - are also
    operationally advantageous, and this should be viewed as an opportunity to
    educate those unaware of the power of batch size reduction.

    http://www.pwc.com/us/en/technology-forecast/2013/issue2/interviews/interview-john-esser.jhtml

    The absence of serious production problems points to the fact that we're
    rolling smaller increments of functionality, and so it's a lot easier to
    make sure the quality level is high.

Even if not everyone around here prefers this style of development, it's
plenty well established.  Figuring out how best to faciliate its use presents
an opportunity to improve Apache, and I firmly believe that our models are
sufficiently flexible to handle it.

Marvin Humphrey

Re: How can we support a faster release cadence?

[Benson Margulies <bi...@gmail.com>]

What I see here is a disagreement about the meaning of the Apache
brand. Some people feel that the Apache brand should always imply a
particular style of project. Other people in the past have written at
length that the ASF should be a big tent that accommodates many styles
of project. There are already project that release frequently,
including the full voting process. It's a lot of work, but they do it.
So it's not true to say that 'community over code' precludes this.
Some communities have chosen this. The discussion at hand is about how
we could make it _easier_.

All the remarks about code quality and stability are, I think, off
topic here. The reason for the release system is legal, not QA. If
people think that more releases, or more automated releases, will
decrease quality, then they shouldn't adopt those patterns in their
communities. The question at hand should be: can we use automation to
make it less labor-intensive to satisfy the legal requirements for an
ASF release. Those requirements are about IP, not about QA>

Re: How can we support a faster release cadence?

[Marvin Humphrey <ma...@rectangular.com>]

On Fri, Feb 7, 2014 at 3:02 AM, Stephen Connolly
<st...@gmail.com> wrote:
> So the second issue is what a PMC member is required to do before voting...
>
> As a PMC member you are required to
>
> 1. Download the source code package
> 2. Compile it as provided
> 3. Test the resulting executable on your own platform
> 4. Verify that the package meets the requirements of the ASF policy on
>    releases
>
> Do we really have to personally do all that *by hand*?

Supervising an autopilot is a fine way to fly a plane, so long as you actually
complete all your checklists and are prepared to seize control when something
unexpected occurs.

Since we're programming computers, we're already using automation.  Adding CI
is just adding more automation, and for functionality testing and platform
compatibility testing, CI is superior to having a handful of PMC members run
manual builds.

For IP review, I think you still need to download the source archive and check
it on your own hardware.  I would want a local tool which...

*   downloads the release candidate
*   generates a list of files added or removed since the last release
*   generates a diff of all changes since the last release
*   verifies that the expanded source archive matches an export from the
    version control tag

I'd then inspect the changes to make sure they passed muster.

*   diff looks legit
*   new files have proper licensing
*   change log up to date
*   etc.

With that kind of tooling, performing review on a weekly release schedule
would not be unduly onerous[1].  A rapid release cadence yields small deltas
which are comparatively easier to inspect.

> Why can we not have a trusted build server hosted on Apache hardware do the
> download of the source package, compile it as provided and run the automated
> acceptance tests (on a range of platforms), the RAT tooling and perhaps
> verify that the source code package matches what is in source control? The
> trusted build server could then report the results to the project mailing
> list and then the PMC members just need to confirm the build server said OK,
> review the commits between the last time they looked at the commits and the
> tag (which they know matches what is in the source bundle) and then vote +1?

I'm concerned about the idea of a "trusted build server".  Central build
servers are extremely attractive targets.

Instead of a "build server", how about just a CI server which is subscribed to
detect release candidates appearing on dist.apache.org?

I think it's might be OK to have a central server create a source release
candidate (to be validated by PMC members on their own hardware in addition to
any validation performed by the server), but binaries are a no-go due to the
prohibitive cost and risk of maintaining secure build servers.  That leaves us
not too far from where we are today, where for many projects an RM supplies
both the source archive and convenience binaries (which they assume the risk
for).

Marvin Humphrey

[1] Assuming everything goes well -- which it won't.  But that's a separate
    email.

Re: How can we support a faster release cadence?

[Andy Seaborne <an...@apache.org>]

On 10/02/14 12:50, Stephen Connolly wrote:
> Well first off, my experience is that users are reluctant to even test
> -alpha- and -beta- releases and consequently there is next to zero chance
> of them testing an RC.
>
> Additionally with fast cadence releases the RC will most likely get
> released anyway... if they have an issue with that *reproducible* build
> they can report it and get the fix in the next RC (which won't be long as
> this is a fast cadence release project)
>
> When I was integrating the Jenkins Credentials support into the Jenkins
> Subversion plugin, we cut a functionally complete release with a -beta-
> label and said: "Look this is ready to go, but we need testing of the
> credentials migration from real users, can you please test it?"
>
> 3 months later we had zero feedback from users.
>
> We've had alpha and RC *releases* of Maven in the past, and we only get
> feedback from users when we cut an actual release.
>
> Users empirically, at least the ones I have seen, do not want to try
> anything with an "RC" or "alpha" or "beta" in the version number. They want
> a solid piece of ground upon which they have a chance of building some
> stability... only if they *really* need a bleeding edge fix/feature will
> they venture to try the RC/alpha/beta
>
> Faster cadence enables more user involvement by getting releases that users
> are willing to consume into their hands faster...

My experience is similar.  Nothing is tested until after a release.
(large) organisational product update cycles drive the testing. The 
testing is months after the release and disconnected from the projects 
release cycle.

Yet we know said org-users have automated test environments and running 
against the codebase or snapshot builds is almost entirely a setup 
exercise on their part.  They would then notice regressions before they 
get baked as a release.

Individuals, start-ups etc use the latest release quite quickly but 
still trailing releases.  Checking anything, including when releases are 
announced as upcoming, does not happen to any significant degree.

Or maybe we release too often with not enough bugs!

</frustration>

	Andy

>
> Does that mean it is right for every project? Hell no... it is a
> trade-off... putting out weekly releases raises other problems...
>
> "I haven't upgraded Jenkins in 4 weeks, what version of Jenkins should I
> upgrade to?" is the most obvious problem...
>
> In other words, when releases are cut every week, users now face the
> problem as to which version they use. Projects need to address that problem
> with things like usage statistics, version rating, recommended versions,
> etc.
>
> A slower release cadence allows for higher quality... but if the version
> you need also has a bug in an unrelated area, you may have no version you
> can choose to use... the fast release cadence gives the user a better
> chance that the feature was added in one version and the bug introduced in
> a later version (and if vice-versa, they can probably back out the bug and
> back-port the feature a lot more easily if they want to roll their own
> build)
>
> As I said at the beginning, I have been working with Jenkins since it's 2nd
> or 3rd public non-sun release... the rapid release cycle worked wonders for
> my engagement... even if at first the sometimes multiple releases per day
> caused issues... the switch to weekly releases was good for Jenkins and I
> think weekly releases have encouraged the Jenkins user community rather
> than hindered it... especially when I compare with the "name-thief fork"
> and it's significantly slower cadence.
>
>
> On 10 February 2014 11:37, Joseph Schaefer <jo...@yahoo.com> wrote:
>
>> I can see why cutting the time window might lead
>> to a faster release cadence, but what I don't get
>> is why this is a desirable goal.  Generally
>> speaking, we try to encourage users to get involved
>> in development, ostensibly by vetting release candidates
>> as a zeroth order thing they can do to help the project.
>>
>> The net effect of reducing opportunities for user participation
>> in the release process would lead me to guess that the opposite
>> flow will happen- that surplussed users will stop following dev
>> and go back to the user lists.  If I'm right, then why
>> is this a good and desirable effect for an Apache project?
>>
>>
>>
>> On Feb 10, 2014, at 3:30 AM, Stephen Connolly <
>> stephen.alan.connolly@gmail.com> wrote:
>>
>>> On Monday, 10 February 2014, Upayavira <uv...@odoko.co.uk> wrote:
>>>
>>>>
>>>>
>>>> On Sun, Feb 9, 2014, at 06:40 AM, Marvin Humphrey wrote:
>>>>> On Sat, Feb 8, 2014 at 3:26 PM, Stephen Connolly
>>>>> <stephen.alan.connolly@gmail.com <javascript:;>> wrote:
>>>>>> 72h for a vote is not a hard and fast rule (you just need a good
>>>> reason for
>>>>>> why you are going shorter and from what I have seen, the board would
>>>>>> probably be ok as long as protections are put in place to safeguard
>> the
>>>>>> community)
>>>>>
>>>>> By now, I think that we've demonstrated in this thread that scheduled
>>>>> votes with a small window (12-24 hours) are practical.
>>>>
>>>> Have we?
>>>>
>>>> I don't believe anyone has expressed the real justification for a 72hr
>>>> window, which is to enable the vote to be *inclusive*. That is,
>>>> inclusive of people who don't live in the same timezone, and who perhaps
>>>> don't work on the codebase full time.
>>>>
>>>> Yes, a 12hr window might make it possible for everyone to have at least
>>>> 4 waking hours in that window, but what if that is your 4hrs of taking
>>>> your kids to school, or cooking dinner for the family. Or if they
>>>> contribute in their spare time, and that 4hrs is whilst they are at
>>>> work. If the project chooses that particular 12hr window as a fixed
>>>> thing, it effectively excludes you from the vote.
>>>
>>>
>>> But this is a *scheduled* vote... If you know that it is something you
>>> *want* to have a chance to vote on, you have sufficient time to ensure
>> the
>>> vote is extended in order for you tiger your say.
>>>
>>> IMHO 72h is needed *when you don't know that there will be a vote*...
>> This
>>> would be a different case... Though I ack that I had only thought this
>> and
>>> not articulated it prior
>>>
>>>
>>>> I am in no way attempting to argue that 72hrs votes is the only way to
>>>> achieve this particular aim, but I do not consider this issue as
>>>> addressed in any way in this thread yet. So:
>>>>
>>>> If we are going to shorten release vote durations, how do we ensure
>>>> inclusivity, both of current, and potential future contributors,
>>>> irrespective of timezone, work pattern, etc?
>>>>
>>>> Upayavira
>>>>
>>>
>>>
>>> --
>>> Sent from my phone
>>
>>
>

Re: How can we support a faster release cadence?

[Stephen Connolly <st...@gmail.com>]

Well first off, my experience is that users are reluctant to even test
-alpha- and -beta- releases and consequently there is next to zero chance
of them testing an RC.

Additionally with fast cadence releases the RC will most likely get
released anyway... if they have an issue with that *reproducible* build
they can report it and get the fix in the next RC (which won't be long as
this is a fast cadence release project)

When I was integrating the Jenkins Credentials support into the Jenkins
Subversion plugin, we cut a functionally complete release with a -beta-
label and said: "Look this is ready to go, but we need testing of the
credentials migration from real users, can you please test it?"

3 months later we had zero feedback from users.

We've had alpha and RC *releases* of Maven in the past, and we only get
feedback from users when we cut an actual release.

Users empirically, at least the ones I have seen, do not want to try
anything with an "RC" or "alpha" or "beta" in the version number. They want
a solid piece of ground upon which they have a chance of building some
stability... only if they *really* need a bleeding edge fix/feature will
they venture to try the RC/alpha/beta

Faster cadence enables more user involvement by getting releases that users
are willing to consume into their hands faster...

Does that mean it is right for every project? Hell no... it is a
trade-off... putting out weekly releases raises other problems...

"I haven't upgraded Jenkins in 4 weeks, what version of Jenkins should I
upgrade to?" is the most obvious problem...

In other words, when releases are cut every week, users now face the
problem as to which version they use. Projects need to address that problem
with things like usage statistics, version rating, recommended versions,
etc.

A slower release cadence allows for higher quality... but if the version
you need also has a bug in an unrelated area, you may have no version you
can choose to use... the fast release cadence gives the user a better
chance that the feature was added in one version and the bug introduced in
a later version (and if vice-versa, they can probably back out the bug and
back-port the feature a lot more easily if they want to roll their own
build)

As I said at the beginning, I have been working with Jenkins since it's 2nd
or 3rd public non-sun release... the rapid release cycle worked wonders for
my engagement... even if at first the sometimes multiple releases per day
caused issues... the switch to weekly releases was good for Jenkins and I
think weekly releases have encouraged the Jenkins user community rather
than hindered it... especially when I compare with the "name-thief fork"
and it's significantly slower cadence.


On 10 February 2014 11:37, Joseph Schaefer <jo...@yahoo.com> wrote:

> I can see why cutting the time window might lead
> to a faster release cadence, but what I don't get
> is why this is a desirable goal.  Generally
> speaking, we try to encourage users to get involved
> in development, ostensibly by vetting release candidates
> as a zeroth order thing they can do to help the project.
>
> The net effect of reducing opportunities for user participation
> in the release process would lead me to guess that the opposite
> flow will happen- that surplussed users will stop following dev
> and go back to the user lists.  If I'm right, then why
> is this a good and desirable effect for an Apache project?
>
>
>
> On Feb 10, 2014, at 3:30 AM, Stephen Connolly <
> stephen.alan.connolly@gmail.com> wrote:
>
> > On Monday, 10 February 2014, Upayavira <uv...@odoko.co.uk> wrote:
> >
> >>
> >>
> >> On Sun, Feb 9, 2014, at 06:40 AM, Marvin Humphrey wrote:
> >>> On Sat, Feb 8, 2014 at 3:26 PM, Stephen Connolly
> >>> <stephen.alan.connolly@gmail.com <javascript:;>> wrote:
> >>>> 72h for a vote is not a hard and fast rule (you just need a good
> >> reason for
> >>>> why you are going shorter and from what I have seen, the board would
> >>>> probably be ok as long as protections are put in place to safeguard
> the
> >>>> community)
> >>>
> >>> By now, I think that we've demonstrated in this thread that scheduled
> >>> votes with a small window (12-24 hours) are practical.
> >>
> >> Have we?
> >>
> >> I don't believe anyone has expressed the real justification for a 72hr
> >> window, which is to enable the vote to be *inclusive*. That is,
> >> inclusive of people who don't live in the same timezone, and who perhaps
> >> don't work on the codebase full time.
> >>
> >> Yes, a 12hr window might make it possible for everyone to have at least
> >> 4 waking hours in that window, but what if that is your 4hrs of taking
> >> your kids to school, or cooking dinner for the family. Or if they
> >> contribute in their spare time, and that 4hrs is whilst they are at
> >> work. If the project chooses that particular 12hr window as a fixed
> >> thing, it effectively excludes you from the vote.
> >
> >
> > But this is a *scheduled* vote... If you know that it is something you
> > *want* to have a chance to vote on, you have sufficient time to ensure
> the
> > vote is extended in order for you tiger your say.
> >
> > IMHO 72h is needed *when you don't know that there will be a vote*...
> This
> > would be a different case... Though I ack that I had only thought this
> and
> > not articulated it prior
> >
> >
> >> I am in no way attempting to argue that 72hrs votes is the only way to
> >> achieve this particular aim, but I do not consider this issue as
> >> addressed in any way in this thread yet. So:
> >>
> >> If we are going to shorten release vote durations, how do we ensure
> >> inclusivity, both of current, and potential future contributors,
> >> irrespective of timezone, work pattern, etc?
> >>
> >> Upayavira
> >>
> >
> >
> > --
> > Sent from my phone
>
>

Re: How can we support a faster release cadence?

[Joseph Schaefer <jo...@yahoo.com>]

I can see why cutting the time window might lead
to a faster release cadence, but what I don’t get
is why this is a desirable goal.  Generally
speaking, we try to encourage users to get involved
in development, ostensibly by vetting release candidates
as a zeroth order thing they can do to help the project.

The net effect of reducing opportunities for user participation
in the release process would lead me to guess that the opposite
flow will happen- that surplussed users will stop following dev
and go back to the user lists.  If I’m right, then why
is this a good and desirable effect for an Apache project?



On Feb 10, 2014, at 3:30 AM, Stephen Connolly <st...@gmail.com> wrote:

> On Monday, 10 February 2014, Upayavira <uv...@odoko.co.uk> wrote:
> 
>> 
>> 
>> On Sun, Feb 9, 2014, at 06:40 AM, Marvin Humphrey wrote:
>>> On Sat, Feb 8, 2014 at 3:26 PM, Stephen Connolly
>>> <stephen.alan.connolly@gmail.com <javascript:;>> wrote:
>>>> 72h for a vote is not a hard and fast rule (you just need a good
>> reason for
>>>> why you are going shorter and from what I have seen, the board would
>>>> probably be ok as long as protections are put in place to safeguard the
>>>> community)
>>> 
>>> By now, I think that we've demonstrated in this thread that scheduled
>>> votes with a small window (12-24 hours) are practical.
>> 
>> Have we?
>> 
>> I don't believe anyone has expressed the real justification for a 72hr
>> window, which is to enable the vote to be *inclusive*. That is,
>> inclusive of people who don't live in the same timezone, and who perhaps
>> don't work on the codebase full time.
>> 
>> Yes, a 12hr window might make it possible for everyone to have at least
>> 4 waking hours in that window, but what if that is your 4hrs of taking
>> your kids to school, or cooking dinner for the family. Or if they
>> contribute in their spare time, and that 4hrs is whilst they are at
>> work. If the project chooses that particular 12hr window as a fixed
>> thing, it effectively excludes you from the vote.
> 
> 
> But this is a *scheduled* vote... If you know that it is something you
> *want* to have a chance to vote on, you have sufficient time to ensure the
> vote is extended in order for you tiger your say.
> 
> IMHO 72h is needed *when you don't know that there will be a vote*... This
> would be a different case... Though I ack that I had only thought this and
> not articulated it prior
> 
> 
>> I am in no way attempting to argue that 72hrs votes is the only way to
>> achieve this particular aim, but I do not consider this issue as
>> addressed in any way in this thread yet. So:
>> 
>> If we are going to shorten release vote durations, how do we ensure
>> inclusivity, both of current, and potential future contributors,
>> irrespective of timezone, work pattern, etc?
>> 
>> Upayavira
>> 
> 
> 
> -- 
> Sent from my phone

Re: How can we support a faster release cadence?

[Stephen Connolly <st...@gmail.com>]

On Monday, 10 February 2014, Upayavira <uv...@odoko.co.uk> wrote:

>
>
> On Sun, Feb 9, 2014, at 06:40 AM, Marvin Humphrey wrote:
> > On Sat, Feb 8, 2014 at 3:26 PM, Stephen Connolly
> > <stephen.alan.connolly@gmail.com <javascript:;>> wrote:
> > > 72h for a vote is not a hard and fast rule (you just need a good
> reason for
> > > why you are going shorter and from what I have seen, the board would
> > > probably be ok as long as protections are put in place to safeguard the
> > > community)
> >
> > By now, I think that we've demonstrated in this thread that scheduled
> > votes with a small window (12-24 hours) are practical.
>
> Have we?
>
> I don't believe anyone has expressed the real justification for a 72hr
> window, which is to enable the vote to be *inclusive*. That is,
> inclusive of people who don't live in the same timezone, and who perhaps
> don't work on the codebase full time.
>
> Yes, a 12hr window might make it possible for everyone to have at least
> 4 waking hours in that window, but what if that is your 4hrs of taking
> your kids to school, or cooking dinner for the family. Or if they
> contribute in their spare time, and that 4hrs is whilst they are at
> work. If the project chooses that particular 12hr window as a fixed
> thing, it effectively excludes you from the vote.


But this is a *scheduled* vote... If you know that it is something you
*want* to have a chance to vote on, you have sufficient time to ensure the
vote is extended in order for you tiger your say.

IMHO 72h is needed *when you don't know that there will be a vote*... This
would be a different case... Though I ack that I had only thought this and
not articulated it prior


> I am in no way attempting to argue that 72hrs votes is the only way to
> achieve this particular aim, but I do not consider this issue as
> addressed in any way in this thread yet. So:
>
> If we are going to shorten release vote durations, how do we ensure
> inclusivity, both of current, and potential future contributors,
> irrespective of timezone, work pattern, etc?
>
> Upayavira
>


-- 
Sent from my phone

Re: How can we support a faster release cadence?

[Upayavira <uv...@odoko.co.uk>]

On Sun, Feb 9, 2014, at 06:40 AM, Marvin Humphrey wrote:
> On Sat, Feb 8, 2014 at 3:26 PM, Stephen Connolly
> <st...@gmail.com> wrote:
> > 72h for a vote is not a hard and fast rule (you just need a good reason for
> > why you are going shorter and from what I have seen, the board would
> > probably be ok as long as protections are put in place to safeguard the
> > community)
> 
> By now, I think that we've demonstrated in this thread that scheduled
> votes with a small window (12-24 hours) are practical.

Have we?

I don't believe anyone has expressed the real justification for a 72hr
window, which is to enable the vote to be *inclusive*. That is,
inclusive of people who don't live in the same timezone, and who perhaps
don't work on the codebase full time.

Yes, a 12hr window might make it possible for everyone to have at least
4 waking hours in that window, but what if that is your 4hrs of taking
your kids to school, or cooking dinner for the family. Or if they
contribute in their spare time, and that 4hrs is whilst they are at
work. If the project chooses that particular 12hr window as a fixed
thing, it effectively excludes you from the vote.

I am in no way attempting to argue that 72hrs votes is the only way to
achieve this particular aim, but I do not consider this issue as
addressed in any way in this thread yet. So:

If we are going to shorten release vote durations, how do we ensure
inclusivity, both of current, and potential future contributors,
irrespective of timezone, work pattern, etc?

Upayavira

Re: How can we support a faster release cadence?

[Marvin Humphrey <ma...@rectangular.com>]

On Sat, Feb 8, 2014 at 3:26 PM, Stephen Connolly
<st...@gmail.com> wrote:
> 72h for a vote is not a hard and fast rule (you just need a good reason for
> why you are going shorter and from what I have seen, the board would
> probably be ok as long as protections are put in place to safeguard the
> community)

By now, I think that we've demonstrated in this thread that scheduled votes
with a small window (12-24 hours) are practical.

Additional tooling could make release review easier, and it would take days or
weeks to set up, not months.  The tooling would be done by the project PMC,
rather than Infra, so we aren't even blocking on additional infrastructure
feature requests -- Jenkins/buildbot and gitpubsub/svnpubsub suffice.

> So the only hard rule per se, is that the PMC must do their IP diligence...

Beyond the legal aspects of making a release, voting is also necessary to
demonstrate that a release has community consensus.

The area we still need to explore, IMO, is what happens when release votes
fail, as that may disrupt the schedule.  In a worst case, a release cycle
might have to be skipped.  Subsequent releases would carry a greater payload
of accumulated work, increasing the bug surface area.

I think we have to accept this possibility.  A community can have a "bias to
ship", but that bias cannot be so strong that it overrides a majority vote
rejecting a release candidate.

Things might spiral out of control if we allowed vetoes on release votes...
but we don't.

If a community is experiencing frequent glitches, that just means
insufficiently mature code is being merged into the mainline too aggressively.
That's easily remedied by dialing back the pace of feature integration.

Marvin Humphrey

Re: How can we support a faster release cadence?

[Stephen Connolly <st...@gmail.com>]

But what if the community wants Apache to have such a rapid cadence?

By forcing the community to decamp to GitHub, then what are we saying to
that community? Sorry we may value community over code, but we value
procedure over both?

If that is the answer, then so be it, let's change our mantra from
"community over code" to "policies and procedures over community over
code"... But that is not the foundation I thought I was joining

There are only some hard and fast rules at the ASF... 72h for a vote is not
a hard and fast rule (you just need a good reason for why you are going
shorter and from what I have seen, the board would probably be ok as long
as protections are put in place to safeguard the community)

So the only hard rule per se, is that the PMC must do their IP diligence...
What I don't want to see is a PMC taking the "easy" way out and pretending
that thy download an check the source bundle for IP... I suspect there are
enough binding votes cast having skimped on that criteria... So we need to
make it easier for the PMC members to do their responsibility and verify
the IP...

Sure we can live in the dark and say that moving faster is not "the Apache
way"... But how does that protect the foundation...

Faster release cadence is about raising the bar in general too... Not just
about helping some projects "dodge" release vote requirements

- Stephen

On Saturday, 8 February 2014, Henri Yandell <he...@yandell.org> wrote:

> If you want to have a faster release cadence, then I think the answer is
> for _you_ (not Apache) to have a faster release cadence. It seems easy to
> look at Apache rules for releases and Linux rules for releases (as an
> underpinning for the features found in Git) and treat them as incompatible,
> but I don't think they should be.
>
> * Go and fork the project code on GitHub.
> * Put your changes in there and PR them up into the Apache codebase.
> * If others want to, they can PR the code to you, and then you can PR the
> code up to the codebase (or the group of you could work as a community
> preparing PRs).
> * The one pushing into the Apache codebase needs to be confident that the
> code is covered by CLAs.
> * You can release in GitHub whenever you want.
> * The Apache release happens less often and follows the rules.
>
> We have some issues to 'defend' against:
>
> * Trademark understanding so that a project's releases on GitHub do not
> appear as being from Apache.
> * Making sure the system is not used for exclusionary reasons (though
> they'll often identify community problems as a root cause anyway).
>
> It's no different than what commercial companies are doing already, either
> by having internal forks or an 'upsell' public version; with both cases
> taking time for patches to work their way back in. Why not have subsets of
> the PMC doing the same?
>
> This takes the legal arguments out and gets us more focused on the 'but a
> singleton community is the apache way' gut unhappiness.
>
> Hen
>


-- 
Sent from my phone

Re: How can we support a faster release cadence?

[Dave Fisher <da...@comcast.net>]

On Feb 11, 2014, at 12:43 PM, sebb wrote:

> On 11 February 2014 17:01, Benson Margulies <bi...@gmail.com> wrote:
>> Could I suggest a focus on making the release process easier? That
>> will benefit everybody, and serve as a platform for ongoing discussion
>> about releases and cadences.
>> 
>> It seems to me that we could make voters' jobs easier. This would help
>> get releases approved _in 72 hours_, to start with.
>> 
>> We ask voters to participate in a business decision by;
>> 
>> 1. being aware of the ongoing work of the community
>> 2. checking the signature
> 
> + hashes
> 
>> 3. building and running enough tests to be willing to endorse that
>> release as a product of the community
> 
> 4. Checking that the NOTICE & LICENSE files are appropriate to the
> release artifacts that contain them.
> 5. Checking that the contents of the source release agree with the SCM
> tag (there should be nothing in the source release that is not in SCM
> - or perhaps directly derived therefrom, but that is unusual)
> 
>> At the start of this thread, someone asked: "If a trusted machine
>> validated the signature, built the package, and ran the tests, could a
>> responsible PMC member vote +1 on the basis of trusting the machine?"
>> After all, all many voters do is to run the tests in the package, and
>> if a someone has committed changes to denature them, the voter might
>> well not notice.
>> 
>> All of this should sit on the platform of my first point above: I
>> submit that a person who has been paying no attention to commits and
>> discussions has no business swooping in and voting on a package based
>> on the other two steps.
> 
> I agree if the vote is +1
> However, I don't think one should disallow reports of packaging bugs
> or test failures - these can be picked up by anyone.
> 
> Also note that whatever is used to check the packaging and release
> should be entirely independent of the packaging mechanism.
> This is necessary to avoid common-mode failures (if that is the correct term).

Exactly.

Here is another scenario to think about. If a PMC bases their votes on Tooling only and not on a careful review of tooling inputs then there can be trouble.

For example. RAT is a great tool, but in checking a release you ought to look into the RAT excludes plus any changes to that file. You then need to manually check those files to make sure it still makes sense to exclude those. RAT excludes can hide inadvertent problems with the release.

The tooling is only as good as its input and it is important to check that during your testing before casting your release vote. The most important -1 on a release is one where there is an IP problem. A PMC that is not pressured by cadence will almost automatically treat even the smallest IP related -1 as a valid VETO. The trouble will be fixed and the release re-spun.

What happens when the cadence is more important to the PMC? The hiccup of a missed cycle is worth it. The community will learn that IP relaly does matter and even tiny infractions are cleaned in a responsibly quick way.

If you are doing it that way and can prove it then the PMC is doing its human best. If anyone calls the PMC on IP or license an established behavior of respect goes along way. Let's be sure to leave the credence intact.

Regards,
Dave

> 
>> 
>> On Tue, Feb 11, 2014 at 11:19 AM, Shane Curcuru <as...@shanecurcuru.org> wrote:
>>> On 2/9/14 2:03 PM, Doug Cutting wrote:
>>>> 
>>>> On Sat, Feb 8, 2014 at 2:44 PM, Henri Yandell <he...@yandell.org> wrote:
>>>>> 
>>>>> * Go and fork the project code on GitHub.
>>>>> * Put your changes in there and PR them up into the Apache codebase.
>>>>> * If others want to, they can PR the code to you, and then you can PR the
>>>>> code up to the codebase (or the group of you could work as a community
>>>>> preparing PRs).
>>>>> * The one pushing into the Apache codebase needs to be confident that the
>>>>> code is covered by CLAs.
>>>>> * You can release in GitHub whenever you want.
>>>>> * The Apache release happens less often and follows the rules.
>>>> 
>>>> 
>>>> Keep in mind that if this is in any way a PMC activity then it is part
>>>> of Apache trying to circumvent the rest of Apache, i.e., not advised.
>>>> A distinct legal entity may indeed fork, re-brand, alter and release
>>>> any Apache project using policies it prefers, but this must be clearly
>>>> separate from any Apache project.  A subset of a PMC acting as
>>>> individuals would be murky territory if they share no common legal
>>>> entity outside Apache.
>>>> 
>>>> Doug
>>> 
>>> 
>>> The key point is: who is the "we" that the world perceives doing this? This
>>> whole discussion really underscores the importance of trademarks and the
>>> Apache brand.
>>> 
>>> We're quite happy for anyone to take our code and ship it just about however
>>> they like.  But they can't call it an Apache project: only a PMC here at
>>> Apache can do that.  While the original reason for most ASF release,
>>> branding, legal, etc. policies is to ensure our legal safety, the very real
>>> effect of these policies and their consistent application in PMCs is that
>>> our projects following these policies are *seen by the rest of the world* as
>>> being "Apache projects".
>>> 
>>> - Shane

Re: How can we support a faster release cadence?

[sebb <se...@gmail.com>]

On 11 February 2014 17:01, Benson Margulies <bi...@gmail.com> wrote:
> Could I suggest a focus on making the release process easier? That
> will benefit everybody, and serve as a platform for ongoing discussion
> about releases and cadences.
>
> It seems to me that we could make voters' jobs easier. This would help
> get releases approved _in 72 hours_, to start with.
>
> We ask voters to participate in a business decision by;
>
> 1. being aware of the ongoing work of the community
> 2. checking the signature

+ hashes

> 3. building and running enough tests to be willing to endorse that
> release as a product of the community

4. Checking that the NOTICE & LICENSE files are appropriate to the
release artifacts that contain them.
5. Checking that the contents of the source release agree with the SCM
tag (there should be nothing in the source release that is not in SCM
- or perhaps directly derived therefrom, but that is unusual)

> At the start of this thread, someone asked: "If a trusted machine
> validated the signature, built the package, and ran the tests, could a
> responsible PMC member vote +1 on the basis of trusting the machine?"
> After all, all many voters do is to run the tests in the package, and
> if a someone has committed changes to denature them, the voter might
> well not notice.
>
> All of this should sit on the platform of my first point above: I
> submit that a person who has been paying no attention to commits and
> discussions has no business swooping in and voting on a package based
> on the other two steps.

I agree if the vote is +1
However, I don't think one should disallow reports of packaging bugs
or test failures - these can be picked up by anyone.

Also note that whatever is used to check the packaging and release
should be entirely independent of the packaging mechanism.
This is necessary to avoid common-mode failures (if that is the correct term).

>
> On Tue, Feb 11, 2014 at 11:19 AM, Shane Curcuru <as...@shanecurcuru.org> wrote:
>> On 2/9/14 2:03 PM, Doug Cutting wrote:
>>>
>>> On Sat, Feb 8, 2014 at 2:44 PM, Henri Yandell <he...@yandell.org> wrote:
>>>>
>>>> * Go and fork the project code on GitHub.
>>>> * Put your changes in there and PR them up into the Apache codebase.
>>>> * If others want to, they can PR the code to you, and then you can PR the
>>>> code up to the codebase (or the group of you could work as a community
>>>> preparing PRs).
>>>> * The one pushing into the Apache codebase needs to be confident that the
>>>> code is covered by CLAs.
>>>> * You can release in GitHub whenever you want.
>>>> * The Apache release happens less often and follows the rules.
>>>
>>>
>>> Keep in mind that if this is in any way a PMC activity then it is part
>>> of Apache trying to circumvent the rest of Apache, i.e., not advised.
>>> A distinct legal entity may indeed fork, re-brand, alter and release
>>> any Apache project using policies it prefers, but this must be clearly
>>> separate from any Apache project.  A subset of a PMC acting as
>>> individuals would be murky territory if they share no common legal
>>> entity outside Apache.
>>>
>>> Doug
>>
>>
>> The key point is: who is the "we" that the world perceives doing this? This
>> whole discussion really underscores the importance of trademarks and the
>> Apache brand.
>>
>> We're quite happy for anyone to take our code and ship it just about however
>> they like.  But they can't call it an Apache project: only a PMC here at
>> Apache can do that.  While the original reason for most ASF release,
>> branding, legal, etc. policies is to ensure our legal safety, the very real
>> effect of these policies and their consistent application in PMCs is that
>> our projects following these policies are *seen by the rest of the world* as
>> being "Apache projects".
>>
>> - Shane

Re: How can we support a faster release cadence?

[Benson Margulies <bi...@gmail.com>]

Could I suggest a focus on making the release process easier? That
will benefit everybody, and serve as a platform for ongoing discussion
about releases and cadences.

It seems to me that we could make voters' jobs easier. This would help
get releases approved _in 72 hours_, to start with.

We ask voters to participate in a business decision by;

1. being aware of the ongoing work of the community
2. checking the signature
3. building and running enough tests to be willing to endorse that
release as a product of the community

At the start of this thread, someone asked: "If a trusted machine
validated the signature, built the package, and ran the tests, could a
responsible PMC member vote +1 on the basis of trusting the machine?"
After all, all many voters do is to run the tests in the package, and
if a someone has committed changes to denature them, the voter might
well not notice.

All of this should sit on the platform of my first point above: I
submit that a person who has been paying no attention to commits and
discussions has no business swooping in and voting on a package based
on the other two steps.


On Tue, Feb 11, 2014 at 11:19 AM, Shane Curcuru <as...@shanecurcuru.org> wrote:
> On 2/9/14 2:03 PM, Doug Cutting wrote:
>>
>> On Sat, Feb 8, 2014 at 2:44 PM, Henri Yandell <he...@yandell.org> wrote:
>>>
>>> * Go and fork the project code on GitHub.
>>> * Put your changes in there and PR them up into the Apache codebase.
>>> * If others want to, they can PR the code to you, and then you can PR the
>>> code up to the codebase (or the group of you could work as a community
>>> preparing PRs).
>>> * The one pushing into the Apache codebase needs to be confident that the
>>> code is covered by CLAs.
>>> * You can release in GitHub whenever you want.
>>> * The Apache release happens less often and follows the rules.
>>
>>
>> Keep in mind that if this is in any way a PMC activity then it is part
>> of Apache trying to circumvent the rest of Apache, i.e., not advised.
>> A distinct legal entity may indeed fork, re-brand, alter and release
>> any Apache project using policies it prefers, but this must be clearly
>> separate from any Apache project.  A subset of a PMC acting as
>> individuals would be murky territory if they share no common legal
>> entity outside Apache.
>>
>> Doug
>
>
> The key point is: who is the "we" that the world perceives doing this? This
> whole discussion really underscores the importance of trademarks and the
> Apache brand.
>
> We're quite happy for anyone to take our code and ship it just about however
> they like.  But they can't call it an Apache project: only a PMC here at
> Apache can do that.  While the original reason for most ASF release,
> branding, legal, etc. policies is to ensure our legal safety, the very real
> effect of these policies and their consistent application in PMCs is that
> our projects following these policies are *seen by the rest of the world* as
> being "Apache projects".
>
> - Shane

Re: How can we support a faster release cadence?

[Doug Cutting <cu...@apache.org>]

On Tue, Feb 11, 2014 at 7:46 PM, Henri Yandell <he...@yandell.org> wrote:

> If we view Apache as a certification of a certain style of quality, then
> that fits nicely with my suggestion that anyone wanting a faster cadence
> should go do such under their own name/brand
>

+1


> and roll the changes up to the mainline for certification.
>

Not sure what "certification" means here.  Submitting changes upstream is
usually a good idea so that the fork can reduce its long-term development
efforts.  As a bonus, improvements are shared with the wider upstream
community, so it's a win-win.


> But the suggestion from Doug is that the act of a subset of the PMC doing
> this would create confusion (I assume that's the murky effect) and be bad.
>

If that subset shares no other common legal entity but Apache then it could
be hard to distinguish from Apache.  If it's clearly the non-Apache effort
of a single individual or corporation then there should be no confusion.
 Perhaps a set of individuals or corporations could somehow make it clear
that they're assuming legal responsibilities and absolve Apache.  What I
wanted to discourage is a (subset of) a PMC thinking they can bypass Apache
policy by working outside of Apache when they don't like Apache policy and
inside when they do.  I thought your suggestion might be read as
encouraging that, even if it's not what you intended, so I sought to
clarify things.  Does that make sense?  Do you disagree?

Doug

Re: How can we support a faster release cadence?

[Henri Yandell <he...@yandell.org>]

On Tue, Feb 11, 2014 at 8:19 AM, Shane Curcuru <as...@shanecurcuru.org> wrote:

> On 2/9/14 2:03 PM, Doug Cutting wrote:
>
>> On Sat, Feb 8, 2014 at 2:44 PM, Henri Yandell <he...@yandell.org> wrote:
>>
>>> * Go and fork the project code on GitHub.
>>> * Put your changes in there and PR them up into the Apache codebase.
>>> * If others want to, they can PR the code to you, and then you can PR the
>>> code up to the codebase (or the group of you could work as a community
>>> preparing PRs).
>>> * The one pushing into the Apache codebase needs to be confident that the
>>> code is covered by CLAs.
>>> * You can release in GitHub whenever you want.
>>> * The Apache release happens less often and follows the rules.
>>>
>>
>> Keep in mind that if this is in any way a PMC activity then it is part
>> of Apache trying to circumvent the rest of Apache, i.e., not advised.
>> A distinct legal entity may indeed fork, re-brand, alter and release
>> any Apache project using policies it prefers, but this must be clearly
>> separate from any Apache project.  A subset of a PMC acting as
>> individuals would be murky territory if they share no common legal
>> entity outside Apache.
>>
>> Doug
>>
>
> The key point is: who is the "we" that the world perceives doing this?
> This whole discussion really underscores the importance of trademarks and
> the Apache brand.
>
> We're quite happy for anyone to take our code and ship it just about
> however they like.  But they can't call it an Apache project: only a PMC
> here at Apache can do that.  While the original reason for most ASF
> release, branding, legal, etc. policies is to ensure our legal safety, the
> very real effect of these policies and their consistent application in PMCs
> is that our projects following these policies are *seen by the rest of the
> world* as being "Apache projects".
>

That's not what Doug implied.

If we view Apache as a certification of a certain style of quality, then
that fits nicely with my suggestion that anyone wanting a faster cadence
should go do such under their own name/brand and roll the changes up to the
mainline for certification. But the suggestion from Doug is that the act of
a subset of the PMC doing this would create confusion (I assume that's the
murky effect) and be bad.

Of course an easy option is to resign from the PMC and remain a committer ;)

Hen

Re: How can we support a faster release cadence?

[Shane Curcuru <as...@shanecurcuru.org>]

On 2/9/14 2:03 PM, Doug Cutting wrote:
> On Sat, Feb 8, 2014 at 2:44 PM, Henri Yandell <he...@yandell.org> wrote:
>> * Go and fork the project code on GitHub.
>> * Put your changes in there and PR them up into the Apache codebase.
>> * If others want to, they can PR the code to you, and then you can PR the
>> code up to the codebase (or the group of you could work as a community
>> preparing PRs).
>> * The one pushing into the Apache codebase needs to be confident that the
>> code is covered by CLAs.
>> * You can release in GitHub whenever you want.
>> * The Apache release happens less often and follows the rules.
>
> Keep in mind that if this is in any way a PMC activity then it is part
> of Apache trying to circumvent the rest of Apache, i.e., not advised.
> A distinct legal entity may indeed fork, re-brand, alter and release
> any Apache project using policies it prefers, but this must be clearly
> separate from any Apache project.  A subset of a PMC acting as
> individuals would be murky territory if they share no common legal
> entity outside Apache.
>
> Doug

The key point is: who is the "we" that the world perceives doing this? 
This whole discussion really underscores the importance of trademarks 
and the Apache brand.

We're quite happy for anyone to take our code and ship it just about 
however they like.  But they can't call it an Apache project: only a PMC 
here at Apache can do that.  While the original reason for most ASF 
release, branding, legal, etc. policies is to ensure our legal safety, 
the very real effect of these policies and their consistent application 
in PMCs is that our projects following these policies are *seen by the 
rest of the world* as being "Apache projects".

- Shane

Re: How can we support a faster release cadence?

[Doug Cutting <cu...@apache.org>]

On Sat, Feb 8, 2014 at 2:44 PM, Henri Yandell <he...@yandell.org> wrote:
> * Go and fork the project code on GitHub.
> * Put your changes in there and PR them up into the Apache codebase.
> * If others want to, they can PR the code to you, and then you can PR the
> code up to the codebase (or the group of you could work as a community
> preparing PRs).
> * The one pushing into the Apache codebase needs to be confident that the
> code is covered by CLAs.
> * You can release in GitHub whenever you want.
> * The Apache release happens less often and follows the rules.

Keep in mind that if this is in any way a PMC activity then it is part
of Apache trying to circumvent the rest of Apache, i.e., not advised.
A distinct legal entity may indeed fork, re-brand, alter and release
any Apache project using policies it prefers, but this must be clearly
separate from any Apache project.  A subset of a PMC acting as
individuals would be murky territory if they share no common legal
entity outside Apache.

Doug

Re: How can we support a faster release cadence?

[Henri Yandell <he...@yandell.org>]

If you want to have a faster release cadence, then I think the answer is
for _you_ (not Apache) to have a faster release cadence. It seems easy to
look at Apache rules for releases and Linux rules for releases (as an
underpinning for the features found in Git) and treat them as incompatible,
but I don't think they should be.

* Go and fork the project code on GitHub.
* Put your changes in there and PR them up into the Apache codebase.
* If others want to, they can PR the code to you, and then you can PR the
code up to the codebase (or the group of you could work as a community
preparing PRs).
* The one pushing into the Apache codebase needs to be confident that the
code is covered by CLAs.
* You can release in GitHub whenever you want.
* The Apache release happens less often and follows the rules.

We have some issues to 'defend' against:

* Trademark understanding so that a project's releases on GitHub do not
appear as being from Apache.
* Making sure the system is not used for exclusionary reasons (though
they'll often identify community problems as a root cause anyway).

It's no different than what commercial companies are doing already, either
by having internal forks or an 'upsell' public version; with both cases
taking time for patches to work their way back in. Why not have subsets of
the PMC doing the same?

This takes the legal arguments out and gets us more focused on the 'but a
singleton community is the apache way' gut unhappiness.

Hen

Re: How can we support a faster release cadence?

[Marvin Humphrey <ma...@rectangular.com>]

On Fri, Feb 7, 2014 at 11:33 AM, Stephen Connolly
<st...@gmail.com> wrote:
> I checked and I think my very original post is reply to dev... Somebody on
> the board list pulled it back in I suspect.

I didn't get my diagnosis quite right.

The copy delivered to dev@community.apache has the following headers:

    To: dev@community.apache.org
    Reply-To: dev@community.apache.org
    Delivered-To: mailing list dev@community.apache.org

The copy delivered to board@apache has the following headers:

    To: dev@community.apache.org
    Reply-To: board@apache.org
    Delivered-To: mailing list board@apache.org

I'm subscribed to both lists, and Gmail has deduped the two mails into one
conversation.  I only see the copy of your mail delivered to board@apache, but
I see all the replies from both the public and private lists intermingled.

I understand the effect you were trying to achieve, but I'm not sure what best
practice is.  Probably sending to the public list and then forwarding to the
private list?

In any case, thanks for starting things off with a very nice email!

Marvin Humphrey

Re: How can we support a faster release cadence?

[Stephen Connolly <st...@gmail.com>]

I checked and I think my very original post is reply to dev... Somebody on
the board list pulled it back in I suspect.

On Friday, 7 February 2014, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:

> Hmm not sure how that happened, board and the "friend" project's private
> list were supposed to be BCC only... Arse!
>
> On Friday, 7 February 2014, Marvin Humphrey <marvin@rectangular.com<javascript:_e(%7B%7D,'cvml','marvin@rectangular.com');>>
> wrote:
>
>> Hi folks,
>>
>> This is an important topic, and when I have more time later I hope to
>> join in the fun.  But for now I'd like to point out that the replies
>> in the kickoff post on dev@community.apache are set to go to
>> board@apache, which is going to make this thread into a real mess of
>> public and private lists.  Please everyone try to keep everything
>> straight.
>>
>> (I would personally prefer that entire discussion take place in public.)
>>
>> Marvin Humphrey
>>
>> On Fri, Feb 7, 2014 at 3:02 AM, Stephen Connolly
>> <st...@gmail.com> wrote:
>> > One of the projects I am involved with is the Jenkins project.
>>
>> ---->8 snip 8<-----
>>
>
>
> --
> Sent from my phone
>


-- 
Sent from my phone

Re: How can we support a faster release cadence?

[Stephen Connolly <st...@gmail.com>]

Hmm not sure how that happened, board and the "friend" project's private
list were supposed to be BCC only... Arse!

On Friday, 7 February 2014, Marvin Humphrey <ma...@rectangular.com> wrote:

> Hi folks,
>
> This is an important topic, and when I have more time later I hope to
> join in the fun.  But for now I'd like to point out that the replies
> in the kickoff post on dev@community.apache are set to go to
> board@apache, which is going to make this thread into a real mess of
> public and private lists.  Please everyone try to keep everything
> straight.
>
> (I would personally prefer that entire discussion take place in public.)
>
> Marvin Humphrey
>
> On Fri, Feb 7, 2014 at 3:02 AM, Stephen Connolly
> <stephen.alan.connolly@gmail.com <javascript:;>> wrote:
> > One of the projects I am involved with is the Jenkins project.
>
> ---->8 snip 8<-----
>


-- 
Sent from my phone

Re: How can we support a faster release cadence?

[Marvin Humphrey <ma...@rectangular.com>]

Hi folks,

This is an important topic, and when I have more time later I hope to
join in the fun.  But for now I'd like to point out that the replies
in the kickoff post on dev@community.apache are set to go to
board@apache, which is going to make this thread into a real mess of
public and private lists.  Please everyone try to keep everything
straight.

(I would personally prefer that entire discussion take place in public.)

Marvin Humphrey

On Fri, Feb 7, 2014 at 3:02 AM, Stephen Connolly
<st...@gmail.com> wrote:
> One of the projects I am involved with is the Jenkins project.

---->8 snip 8<-----

Re: How can we support a faster release cadence?

[Brian LeRoux <b...@brian.io>]

Well, here's an 'doing it live' example for you. This is the parent issue
for our next release:

https://issues.apache.org/jira/browse/CB-5854

Its only adding one more step... which is or isn't a big deal depending on
how you look at it. (We have added that step btw.) Sometimes we ship
multiple times a month. Thats over 50 issues to close to do that. It's like
water torture; adding a drop isn't a big deal in itself but in aggregate it
will bore through your skull.

Working at Adobe I've learned a great deal about how 'just one more step'
isn't a big deal. We're not exactly characterized as nimble, effective,
efficient, or lean. That, to me, certainly is big deal.

Anyhow, this discussion most certainly has a foggy cultural aspect to
contrast with the direct returns in quality and predictability.





On Fri, Feb 14, 2014 at 1:23 PM, Niall Pemberton
<ni...@gmail.com>wrote:

> I love the release cadence - more projects should do it - and I can
> understand why you fight anything that you believe would hinder that. But I
> don't really get how a vote affects that - whatever cycle you're on - it
> just shifts it 3 days? So if you were cutting a release on the last Friday
> of every month (for example) - nothing changes for the user except that the
> monthly release is available 3 days later - they still get it monthly, just
> on the Monday?
>
> Niall
>
>
> On Thu, Feb 13, 2014 at 3:25 AM, Brian LeRoux <b...@brian.io> wrote:
>
>> I'd like to throw out some thoughts in support of this thinking and help
>> explore how we can support faster releases at Apache.
>>
>> Cordova has bias to shipping. We started shipping on a schedule mid 2011
>> and this was a very deliberate choice, after two years of scattered, and
>> frankly reactionary, releases.
>>
>> At that time we called the project PhoneGap and we realized our offering
>> was playing cat and mouse with the very fast moving dependencies of iOS and
>> Android. Being reactionary made shipping a fire drill, inevitably drawn out
>> since we didn't exercise those muscles enough, and ultimately this made our
>> software a risk for adoption. We didn't want to be a risk for adoption. We
>> also did not want our volunteer committership killing themselves every time
>> iOS or Android landed a patch.
>>
>> Moving to a schedule acted as a forcing function to exercise those
>> muscles, find our cadence, and only positives to the code and community
>> resulted. Shipping brought our core together. It meant if we didn't have a
>> fix for a feature the branch would land in the next release which is only a
>> month away. This built huge confidence in our team by our community. Our
>> code become better tested, and more streamlined. A consistent release
>> cadence not only helped us find more quality in our code, but that
>> confidence really helped us build our committer and developer community.
>> The story is hardly unique: Chrome, Ubuntu, Docker, Firefox, and many
>> others have adopted this model in recent years.
>>
>> I feel anything that can be considered a better practice for higher
>> quality code and driving community confidence, and subsequently adoption,
>> really embodies Apache ideals.
>>
>> The current process could be largely automated and the vote doesn't
>> necessarily have to be in the form of an email. I've found these past weeks
>> the act of voting seems near cultural at Apache so I hope that doesn't
>> sound crazy! I mean well.
>>
>> Another issue I am personally unclear on is the wide variety of artifacts
>> destinations that an Apache project can be shipped today. Maven has some of
>> these smarts built in but projects like the npm registry do not. Another
>> area we need to address is the proliferation of various app stores. I'm not
>> a fan of them, but they happen, and we should have a mechanism for our
>> projects to deliver to them.
>>
>>
>> On Fri, Feb 7, 2014 at 3:02 AM, Stephen Connolly <
>> stephen.alan.connolly@gmail.com> wrote:
>>
>>> One of the projects I am involved with is the Jenkins project. At
>>> Jenkins we cut a release of Jenkins every wednesday... assuming the test
>>> all pass...
>>> Not every release is as stable as people might desire (the tests don't
>>> catch everything), hence there is the LTS release line, but none the less,
>>> there is a major release every 7 days... and if you look at the usage stats
>>> (e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg)
>>> most users actually stick fairly close to the latest release.
>>>
>>> I have found that this 7 day release cadence can be really helpful for
>>> some code bases.
>>>
>>> When I started to think about could we follow this model for the Maven
>>> project as we move towards Maven 4.0, there is one thing that gets in the
>>> way... namely release votes.
>>>
>>> The standard answer is that we could publish snapshots... but those are
>>> not indented for use by users... and where the cadence can help is that
>>> these things can be picked up by users.
>>>
>>> So what is it that gets in the way with release votes:
>>>
>>> * The 72h "soft" requirement for vote duration
>>>
>>> * The actions that a PMC member is required to perform before they can
>>> vote. See http://www.apache.org/dev/release which states:
>>>
>>>     > Before voting +1 PMC members are required to download the signed
>>> source code package, compile it as provided, and test the resulting
>>> executable on their own platform, along with also verifying that the
>>> package meets the requirements of the ASF policy on releases.
>>>
>>> So how exactly do these things get in the way?
>>>
>>> Well as I see it the 72h vote duration isn't necessarily a big deal...
>>> we need some duration of notice about what is going into the release, there
>>> will always be people who feel the duration is either too short or two
>>> long... but with a 7 day cadence and maybe a few hours before the release
>>> manager closes out the vote and then you wait for the release to finished
>>> syncing to the mirrors and then the release manager gets a chance to verify
>>> that the release has synced to at least one mirror... you could easily lose
>>> half a day's duration in that process... oh look the release is out 3.5
>>> days after it was cut... and we're cutting another one in 3.5 days... it is
>>> likely we will not get much meaningful feedback from users in the remaining
>>> 3.5 days... so essentially you end up with a ping-pong of break... skip...
>>> fix since if a bleeding edge user finds an issue in 4.0.56 we will have cut
>>> 4.0.57 by the time they report it to us and the fix ends up in 4.0.58...
>>> with a shorter vote duration, say 12h, the bleeding edge user reports the
>>> issue, we fix and the next release is the one they can use.
>>>
>>> In the context of a fast cadence, where every committer in the community
>>> knows there will be a release on wednesday cut from the last revision that
>>> passed all the tests on the CI system unless there have been no commits
>>> since the last release that meet that criteria, do we need to wait the full
>>> 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast
>>> during those 12h... and if not, well just extend until enough votes are
>>> cast)
>>>
>>> I think this is different use case from my understanding of the concerns
>>> that drove the 72h vote duration convention, as this would not be 3 PMC
>>> members who all work for the same company and are in the same location
>>> conspiring to drive their changes into the release... everything would be
>>> happening in the open and a 12h window mid-week should allow at least 4h of
>>> waking time in any TZ.
>>>
>>> So the second issue is what a PMC member is required to do before
>>> voting...
>>>
>>> As a PMC member you are required to
>>>
>>> 1. Download the source code package
>>>  2. Compile it as provided
>>> 3. Test the resulting executable on your own platform
>>> 4. Verify that the package meets the requirements of the ASF policy on
>>> releases
>>>
>>> Do we really have to personally do all that *by hand*?
>>>
>>> Why can we not have a trusted build server hosted on Apache hardware do
>>> the download of the source package, compile it as provided and run the
>>> automated acceptance tests (on a range of platforms), the RAT tooling and
>>> perhaps verify that the source code package matches what is in source
>>> control? The trusted build server could then report the results to the
>>> project mailing list and then the PMC members just need to confirm the
>>> build server said OK, review the commits between the last time they looked
>>> at the commits and the tag (which they know matches what is in the source
>>> bundle) and then vote +1?
>>>
>>> The PMC members are supposed to be keeping an eye on the commits anyway,
>>> so that shouldn't be too onerous, and the release manager could even
>>> provide a link to the build server confirmation build in the VOTE email.
>>>
>>> I would appreciate any people's thoughts on the above.
>>>
>>> -Stephen
>>>
>>> P.S.
>>> * Speaking in my personal capacity as a member of the ASF.
>>>  * I am not saying that Maven will move to such a model, or even wants
>>> to move to such a model... more that I was thinking about the issues that
>>> might prevent us if we so desired... I know other projects at Apache are
>>> interested in fast release cadence however, so getting this topic discussed
>>> in the open is no bad thing IMHO
>>>
>>>
>>
>

Re: How can we support a faster release cadence?

[Niall Pemberton <ni...@gmail.com>]

I love the release cadence - more projects should do it - and I can
understand why you fight anything that you believe would hinder that. But I
don't really get how a vote affects that - whatever cycle you're on - it
just shifts it 3 days? So if you were cutting a release on the last Friday
of every month (for example) - nothing changes for the user except that the
monthly release is available 3 days later - they still get it monthly, just
on the Monday?

Niall


On Thu, Feb 13, 2014 at 3:25 AM, Brian LeRoux <b...@brian.io> wrote:

> I'd like to throw out some thoughts in support of this thinking and help
> explore how we can support faster releases at Apache.
>
> Cordova has bias to shipping. We started shipping on a schedule mid 2011
> and this was a very deliberate choice, after two years of scattered, and
> frankly reactionary, releases.
>
> At that time we called the project PhoneGap and we realized our offering
> was playing cat and mouse with the very fast moving dependencies of iOS and
> Android. Being reactionary made shipping a fire drill, inevitably drawn out
> since we didn't exercise those muscles enough, and ultimately this made our
> software a risk for adoption. We didn't want to be a risk for adoption. We
> also did not want our volunteer committership killing themselves every time
> iOS or Android landed a patch.
>
> Moving to a schedule acted as a forcing function to exercise those
> muscles, find our cadence, and only positives to the code and community
> resulted. Shipping brought our core together. It meant if we didn't have a
> fix for a feature the branch would land in the next release which is only a
> month away. This built huge confidence in our team by our community. Our
> code become better tested, and more streamlined. A consistent release
> cadence not only helped us find more quality in our code, but that
> confidence really helped us build our committer and developer community.
> The story is hardly unique: Chrome, Ubuntu, Docker, Firefox, and many
> others have adopted this model in recent years.
>
> I feel anything that can be considered a better practice for higher
> quality code and driving community confidence, and subsequently adoption,
> really embodies Apache ideals.
>
> The current process could be largely automated and the vote doesn't
> necessarily have to be in the form of an email. I've found these past weeks
> the act of voting seems near cultural at Apache so I hope that doesn't
> sound crazy! I mean well.
>
> Another issue I am personally unclear on is the wide variety of artifacts
> destinations that an Apache project can be shipped today. Maven has some of
> these smarts built in but projects like the npm registry do not. Another
> area we need to address is the proliferation of various app stores. I'm not
> a fan of them, but they happen, and we should have a mechanism for our
> projects to deliver to them.
>
>
> On Fri, Feb 7, 2014 at 3:02 AM, Stephen Connolly <
> stephen.alan.connolly@gmail.com> wrote:
>
>> One of the projects I am involved with is the Jenkins project. At Jenkins
>> we cut a release of Jenkins every wednesday... assuming the test all pass...
>> Not every release is as stable as people might desire (the tests don't
>> catch everything), hence there is the LTS release line, but none the less,
>> there is a major release every 7 days... and if you look at the usage stats
>> (e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg)
>> most users actually stick fairly close to the latest release.
>>
>> I have found that this 7 day release cadence can be really helpful for
>> some code bases.
>>
>> When I started to think about could we follow this model for the Maven
>> project as we move towards Maven 4.0, there is one thing that gets in the
>> way... namely release votes.
>>
>> The standard answer is that we could publish snapshots... but those are
>> not indented for use by users... and where the cadence can help is that
>> these things can be picked up by users.
>>
>> So what is it that gets in the way with release votes:
>>
>> * The 72h "soft" requirement for vote duration
>>
>> * The actions that a PMC member is required to perform before they can
>> vote. See http://www.apache.org/dev/release which states:
>>
>>     > Before voting +1 PMC members are required to download the signed
>> source code package, compile it as provided, and test the resulting
>> executable on their own platform, along with also verifying that the
>> package meets the requirements of the ASF policy on releases.
>>
>> So how exactly do these things get in the way?
>>
>> Well as I see it the 72h vote duration isn't necessarily a big deal... we
>> need some duration of notice about what is going into the release, there
>> will always be people who feel the duration is either too short or two
>> long... but with a 7 day cadence and maybe a few hours before the release
>> manager closes out the vote and then you wait for the release to finished
>> syncing to the mirrors and then the release manager gets a chance to verify
>> that the release has synced to at least one mirror... you could easily lose
>> half a day's duration in that process... oh look the release is out 3.5
>> days after it was cut... and we're cutting another one in 3.5 days... it is
>> likely we will not get much meaningful feedback from users in the remaining
>> 3.5 days... so essentially you end up with a ping-pong of break... skip...
>> fix since if a bleeding edge user finds an issue in 4.0.56 we will have cut
>> 4.0.57 by the time they report it to us and the fix ends up in 4.0.58...
>> with a shorter vote duration, say 12h, the bleeding edge user reports the
>> issue, we fix and the next release is the one they can use.
>>
>> In the context of a fast cadence, where every committer in the community
>> knows there will be a release on wednesday cut from the last revision that
>> passed all the tests on the CI system unless there have been no commits
>> since the last release that meet that criteria, do we need to wait the full
>> 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast
>> during those 12h... and if not, well just extend until enough votes are
>> cast)
>>
>> I think this is different use case from my understanding of the concerns
>> that drove the 72h vote duration convention, as this would not be 3 PMC
>> members who all work for the same company and are in the same location
>> conspiring to drive their changes into the release... everything would be
>> happening in the open and a 12h window mid-week should allow at least 4h of
>> waking time in any TZ.
>>
>> So the second issue is what a PMC member is required to do before
>> voting...
>>
>> As a PMC member you are required to
>>
>> 1. Download the source code package
>>  2. Compile it as provided
>> 3. Test the resulting executable on your own platform
>> 4. Verify that the package meets the requirements of the ASF policy on
>> releases
>>
>> Do we really have to personally do all that *by hand*?
>>
>> Why can we not have a trusted build server hosted on Apache hardware do
>> the download of the source package, compile it as provided and run the
>> automated acceptance tests (on a range of platforms), the RAT tooling and
>> perhaps verify that the source code package matches what is in source
>> control? The trusted build server could then report the results to the
>> project mailing list and then the PMC members just need to confirm the
>> build server said OK, review the commits between the last time they looked
>> at the commits and the tag (which they know matches what is in the source
>> bundle) and then vote +1?
>>
>> The PMC members are supposed to be keeping an eye on the commits anyway,
>> so that shouldn't be too onerous, and the release manager could even
>> provide a link to the build server confirmation build in the VOTE email.
>>
>> I would appreciate any people's thoughts on the above.
>>
>> -Stephen
>>
>> P.S.
>> * Speaking in my personal capacity as a member of the ASF.
>>  * I am not saying that Maven will move to such a model, or even wants to
>> move to such a model... more that I was thinking about the issues that
>> might prevent us if we so desired... I know other projects at Apache are
>> interested in fast release cadence however, so getting this topic discussed
>> in the open is no bad thing IMHO
>>
>>
>

Re: How can we support a faster release cadence?

[Dave Cottlehuber <dc...@jsonified.com>]

On 13 February 2014 04:25, Brian LeRoux <b...@brian.io> wrote:
> I'd like to throw out some thoughts in support of this thinking and help
> explore how we can support faster releases at Apache.
>
> Cordova has bias to shipping. We started shipping on a schedule mid 2011
> and this was a very deliberate choice, after two years of scattered, and
> frankly reactionary, releases.
>
> At that time we called the project PhoneGap and we realized our offering
> was playing cat and mouse with the very fast moving dependencies of iOS and
> Android. Being reactionary made shipping a fire drill, inevitably drawn out
> since we didn't exercise those muscles enough, and ultimately this made our
> software a risk for adoption. We didn't want to be a risk for adoption. We
> also did not want our volunteer committership killing themselves every time
> iOS or Android landed a patch.
>
> Moving to a schedule acted as a forcing function to exercise those muscles,
> find our cadence, and only positives to the code and community
> resulted. Shipping brought our core together. It meant if we didn't have a
> fix for a feature the branch would land in the next release which is only a
> month away. This built huge confidence in our team by our community. Our
> code become better tested, and more streamlined. A consistent release
> cadence not only helped us find more quality in our code, but that
> confidence really helped us build our committer and developer community.
> The story is hardly unique: Chrome, Ubuntu, Docker, Firefox, and many
> others have adopted this model in recent years.

I agree; in the CouchDB community we had a similar experience. Addressing it
has been positive both for our brand and for our community. It's also been
a triumph of Apache values of community over code, demonstrating that the
incubator process, as well as addressing legal concerns via due diligence,
is also capable of sustaining communities who can survive the acrimonious
departure of the founder. Moving to a time-driven release has helped us
enormously.

> I feel anything that can be considered a better practice for higher quality
> code and driving community confidence, and subsequently adoption, really
> embodies Apache ideals.

The faster our code is distributed, the faster we get feedback, as Stephen's
also said.

> The current process could be largely automated and the vote doesn't
> necessarily have to be in the form of an email. I've found these past weeks
> the act of voting seems near cultural at Apache so I hope that doesn't
> sound crazy! I mean well.
>
> Another issue I am personally unclear on is the wide variety of artifacts
> destinations that an Apache project can be shipped today. Maven has some of
> these smarts built in but projects like the npm registry do not. Another
> area we need to address is the proliferation of various app stores. I'm not
> a fan of them, but they happen, and we should have a mechanism for our
> projects to deliver to them.
>
>
> On Fri, Feb 7, 2014 at 3:02 AM, Stephen Connolly <
> stephen.alan.connolly@gmail.com> wrote:
...
>> So what is it that gets in the way with release votes:
>>
>> * The 72h "soft" requirement for vote duration
>>
>> * The actions that a PMC member is required to perform before they can
>> vote. See http://www.apache.org/dev/release which states:
>>
>>     > Before voting +1 PMC members are required to download the signed
>> source code package, compile it as provided, and test the resulting
>> executable on their own platform, along with also verifying that the
>> package meets the requirements of the ASF policy on releases.

This last piece is important - I'll bring it up later on.

>> So how exactly do these things get in the way?
>>
>> Well as I see it the 72h vote duration isn't necessarily a big deal... we
>> need some duration of notice about what is going into the release, there
>> will always be people who feel the duration is either too short or two
>> long... but with a 7 day cadence and maybe a few hours before the release
>> manager closes out the vote and then you wait for the release to finished
>> syncing to the mirrors and then the release manager gets a chance to verify
>> that the release has synced to at least one mirror... you could easily lose
>> half a day's duration in that process... oh look the release is out 3.5
>> days after it was cut... and we're cutting another one in 3.5 days... it is
>> likely we will not get much meaningful feedback from users in the remaining
>> 3.5 days... so essentially you end up with a ping-pong of break... skip...
>> fix since if a bleeding edge user finds an issue in 4.0.56 we will have cut
>> 4.0.57 by the time they report it to us and the fix ends up in 4.0.58...
>> with a shorter vote duration, say 12h, the bleeding edge user reports the
>> issue, we fix and the next release is the one they can use.

Surely 27 hours is a *guidance* not a law. If a project's community wants to
run fast then presumably they also have the commit rate to drive this, and a
diversity across the community, committers & PMC to feel that this isn't Evil
Co sneaking in their cronies. I'm sure that a rogue release or unbalanced
PMC will get appropriate board attention in the mid term, if not the short term.

>> In the context of a fast cadence, where every committer in the community
>> knows there will be a release on wednesday cut from the last revision that
>> passed all the tests on the CI system unless there have been no commits
>> since the last release that meet that criteria, do we need to wait the full
>> 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast
>> during those 12h... and if not, well just extend until enough votes are
>> cast)
>>
>> I think this is different use case from my understanding of the concerns
>> that drove the 72h vote duration convention, as this would not be 3 PMC
>> members who all work for the same company and are in the same location
>> conspiring to drive their changes into the release... everything would be
>> happening in the open and a 12h window mid-week should allow at least 4h of
>> waking time in any TZ.

Personally (as a volunteer committer) that feels tight, but again that's my
commitment for CouchDB, and not Cordova or Jenkins or whatever. If we can
respect the community concensus then why not a shorter window? The reality
is that very few changes/patches land immediately in master 30 seconds before
the release, and part of being a PMC member is giving trust to your committers
to DTRT in your absence. And if we are releasing weekly then the delta of trust
is actually pretty low.

Personally, without a full history of the ASF, I don't yet see a
specific, concrete
example of how this is going to expose the Foundation and/or the Brand.

BTW I'm at IETF in London, feel free to school me there in person - I'd love
to hear your war stories.

>> So the second issue is what a PMC member is required to do before voting...
>>
>> As a PMC member you are required to
>>
>> 1. Download the source code package
>> 2. Compile it as provided
>> 3. Test the resulting executable on your own platform
>> 4. Verify that the package meets the requirements of the ASF policy on
>> releases
>>
>> Do we really have to personally do all that *by hand*?

Agreed. I'm not convinced that building it by hand, as a PMC member,
is magically
going to improve the reliability. In fact, if each PMC member or
community member
sets up a simple test bot or CI on their preferred platform(s)
arguably that's better time
spent, and a more reliable result as it can be run after every commit,
or even before
git merging to the release / master branch.

However, to pick up the point "package meets the requirements of the ASF policy
on releases." from earlier, I don't see how a bot or script can
scrutinise a list of
commits or diffs, and make a reasoned decision that that code meets ASF
requirements from an IP clearance. I might not have phrased that
perfectly, but it's
one important point that I wasn't aware of until this discussion
brewed. Are there
other points that should be on Brian & my list?

>> Why can we not have a trusted build server hosted on Apache hardware do
>> the download of the source package, compile it as provided and run the
>> automated acceptance tests (on a range of platforms), the RAT tooling and
>> perhaps verify that the source code package matches what is in source
>> control? The trusted build server could then report the results to the
>> project mailing list and then the PMC members just need to confirm the
>> build server said OK, review the commits between the last time they looked
>> at the commits and the tag (which they know matches what is in the source
>> bundle) and then vote +1?

I thought this is what the ASF build farm is supposed to enable, surely?

>> The PMC members are supposed to be keeping an eye on the commits anyway,
>> so that shouldn't be too onerous, and the release manager could even
>> provide a link to the build server confirmation build in the VOTE email.
>>
>> I would appreciate any people's thoughts on the above.
>>
>> -Stephen

In addition to Phil & Benson's pithy and timely summaries,

- is there a specific example of how faster *releases* have exposed
the ASF in the past?

- is there some legal finesse that exposes committers or PMC members
if weekly tarballs,
but not full releases, are made available through some automated
process? nightly builds
are de rigeur nowadays.

- for long term ASF members, is the release cadence something that you
*feel* is a per-
project decison, assuming the above concerns are taken care of?

Hopefully that nails down facts WRT to legal constraints, and also
feeling WRT the ASF community.

A+
Dave

Re: How can we support a faster release cadence?

[Brian LeRoux <b...@brian.io>]

Hey Alex, some answers inline:



> But what is Cordova's plan if a major defect is found too late for the
> "schedule"?
>

We use branches so cherry picking features in, and out, of release branches
happens on occasion. Its pretty easy and clean w/ git to do this.



> Would you skip a release?
>

No. Though we have delayed them. And that happens fairly often. We're not
crazy restrictive about the timing. We *aim* for 10 MINOR releases (with 10
companion rc candidates) and 1 MAJOR a year. Other projects, with larger
communities than ours, have to segment into separate channels on a 6 week
cadence. I could see Cordova getting there someday if this mobile thing
continues to grow.

This is just talking about our platforms. We have other assets like plugins
and our command line interface which can see weekly incremental updates. In
other realities I participate in we ship daily. Revision control,
continuous integration, and rigor in testing make this sort of thing not
only possible but financially desirable for commercial web properties.



> And what if something more subjective comes up like a some user showing up
> at the last minute and saying they don't like the usability or name of
> something and a grass roots movement grows around it.
>

Oh, those sorts of things happen and are welcome. If its good we'll do it.
If it doesn't fit we won't. That diversity makes open source the better
model, to me.

Re: How can we support a faster release cadence?

[Alex Harui <ah...@adobe.com>]

My (more than) 2 cents on this topic.  It's going to be really interesting to see how the board decides on this sort of thing.

Your plan is completely logical.  Plenty of business operate this way like newspapers and car manufacturers.  Everybody knows when the next "release" is.  Car manufacturers release next year's models before the absolute end of the year maybe to buy some wiggle room if there is a major defect found, but there is no skipping a year.  And newspapers can't really skip a day either.  So they ship with hopefully minor bugs and newspapers print corrections and car manufacturers recall your car.  I remember reading once that 97% of all cars have been recalled for some issue.

And I get how having a schedule helps coordinate the core volunteers.

But what is Cordova's plan if a major defect is found too late for the "schedule"?  Would you skip a release?  And what if something more subjective comes up like a some user showing up at the last minute and saying they don't like the usability or name of something and a grass roots movement grows around it.

Somewhere along the way, I picked up the notion that the 72 hour vote was for the non-core.  That Apache communities have folks who are busy with their day jobs and can't always get to respond in 24 hours.  Several folks have alluded to this in the various threads, but nobody has said that it is "policy".

If you are a house builder, I don't think you can buy a piece of property and start building a house.  I'm pretty sure you have to post Public Notice signs on the property and in the paper and hold a public hearing about whether you can use the land to build the house you want to build.  Around where I live, I rarely ever see anything get fully rejected, and there seems to be a faction that wants to fight any development and maybe this process is just so they can feel heard, but it seems to be the process.  My sense is that Apache wants us to work this way, but the question is whether we "must".

Having a four day manufacturing pipeline is definitely problematic (3 days for voting, 1 day for mirror propagation) for rapid releases, especially since many other open source organizations can ship as soon as it is checked in, but maybe this is also part of the Apache Way.  We apply an extra coat of paint, or do additional safety checks, or cater to the community members who can't be as active because their working on the project isn't their main job.

Thanks for reading,
-Alex


From: Brian LeRoux <b@...@brian.io>>
Reply-To: "board@apache.org<ma...@apache.org>" <bo...@apache.org>>
Date: Wednesday, February 12, 2014 7:25 PM
To: Apache Board <bo...@apache.org>>
Cc: "dev@community.apache.org<ma...@community.apache.org>" <de...@community.apache.org>>
Subject: Re: How can we support a faster release cadence?

I'd like to throw out some thoughts in support of this thinking and help explore how we can support faster releases at Apache.

Cordova has bias to shipping. We started shipping on a schedule mid 2011 and this was a very deliberate choice, after two years of scattered, and frankly reactionary, releases.

At that time we called the project PhoneGap and we realized our offering was playing cat and mouse with the very fast moving dependencies of iOS and Android. Being reactionary made shipping a fire drill, inevitably drawn out since we didn't exercise those muscles enough, and ultimately this made our software a risk for adoption. We didn't want to be a risk for adoption. We also did not want our volunteer committership killing themselves every time iOS or Android landed a patch.

Moving to a schedule acted as a forcing function to exercise those muscles, find our cadence, and only positives to the code and community resulted. Shipping brought our core together. It meant if we didn't have a fix for a feature the branch would land in the next release which is only a month away. This built huge confidence in our team by our community. Our code become better tested, and more streamlined. A consistent release cadence not only helped us find more quality in our code, but that confidence really helped us build our committer and developer community. The story is hardly unique: Chrome, Ubuntu, Docker, Firefox, and many others have adopted this model in recent years.

I feel anything that can be considered a better practice for higher quality code and driving community confidence, and subsequently adoption, really embodies Apache ideals.

The current process could be largely automated and the vote doesn't necessarily have to be in the form of an email. I've found these past weeks the act of voting seems near cultural at Apache so I hope that doesn't sound crazy! I mean well.

Another issue I am personally unclear on is the wide variety of artifacts destinations that an Apache project can be shipped today. Maven has some of these smarts built in but projects like the npm registry do not. Another area we need to address is the proliferation of various app stores. I'm not a fan of them, but they happen, and we should have a mechanism for our projects to deliver to them.


On Fri, Feb 7, 2014 at 3:02 AM, Stephen Connolly <st...@gmail.com>> wrote:
One of the projects I am involved with is the Jenkins project. At Jenkins we cut a release of Jenkins every wednesday... assuming the test all pass...
Not every release is as stable as people might desire (the tests don't catch everything), hence there is the LTS release line, but none the less, there is a major release every 7 days... and if you look at the usage stats (e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg) most users actually stick fairly close to the latest release.

I have found that this 7 day release cadence can be really helpful for some code bases.

When I started to think about could we follow this model for the Maven project as we move towards Maven 4.0, there is one thing that gets in the way... namely release votes.

The standard answer is that we could publish snapshots... but those are not indented for use by users... and where the cadence can help is that these things can be picked up by users.

So what is it that gets in the way with release votes:

* The 72h "soft" requirement for vote duration

* The actions that a PMC member is required to perform before they can vote. See http://www.apache.org/dev/release which states:

    > Before voting +1 PMC members are required to download the signed source code package, compile it as provided, and test the resulting executable on their own platform, along with also verifying that the package meets the requirements of the ASF policy on releases.

So how exactly do these things get in the way?

Well as I see it the 72h vote duration isn't necessarily a big deal... we need some duration of notice about what is going into the release, there will always be people who feel the duration is either too short or two long... but with a 7 day cadence and maybe a few hours before the release manager closes out the vote and then you wait for the release to finished syncing to the mirrors and then the release manager gets a chance to verify that the release has synced to at least one mirror... you could easily lose half a day's duration in that process... oh look the release is out 3.5 days after it was cut... and we're cutting another one in 3.5 days... it is likely we will not get much meaningful feedback from users in the remaining 3.5 days... so essentially you end up with a ping-pong of break... skip... fix since if a bleeding edge user finds an issue in 4.0.56 we will have cut 4.0.57 by the time they report it to us and the fix ends up in 4.0.58... with a shorter vote duration, say 12h, the bleeding edge user reports the issue, we fix and the next release is the one they can use.

In the context of a fast cadence, where every committer in the community knows there will be a release on wednesday cut from the last revision that passed all the tests on the CI system unless there have been no commits since the last release that meet that criteria, do we need to wait the full 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast during those 12h... and if not, well just extend until enough votes are cast)

I think this is different use case from my understanding of the concerns that drove the 72h vote duration convention, as this would not be 3 PMC members who all work for the same company and are in the same location conspiring to drive their changes into the release... everything would be happening in the open and a 12h window mid-week should allow at least 4h of waking time in any TZ.

So the second issue is what a PMC member is required to do before voting...

As a PMC member you are required to

1. Download the source code package
2. Compile it as provided
3. Test the resulting executable on your own platform
4. Verify that the package meets the requirements of the ASF policy on releases

Do we really have to personally do all that *by hand*?

Why can we not have a trusted build server hosted on Apache hardware do the download of the source package, compile it as provided and run the automated acceptance tests (on a range of platforms), the RAT tooling and perhaps verify that the source code package matches what is in source control? The trusted build server could then report the results to the project mailing list and then the PMC members just need to confirm the build server said OK, review the commits between the last time they looked at the commits and the tag (which they know matches what is in the source bundle) and then vote +1?

The PMC members are supposed to be keeping an eye on the commits anyway, so that shouldn't be too onerous, and the release manager could even provide a link to the build server confirmation build in the VOTE email.

I would appreciate any people's thoughts on the above.

-Stephen

P.S.
* Speaking in my personal capacity as a member of the ASF.
* I am not saying that Maven will move to such a model, or even wants to move to such a model... more that I was thinking about the issues that might prevent us if we so desired... I know other projects at Apache are interested in fast release cadence however, so getting this topic discussed in the open is no bad thing IMHO

Re: How can we support a faster release cadence?

[Brian LeRoux <b...@brian.io>]

I'd like to throw out some thoughts in support of this thinking and help
explore how we can support faster releases at Apache.

Cordova has bias to shipping. We started shipping on a schedule mid 2011
and this was a very deliberate choice, after two years of scattered, and
frankly reactionary, releases.

At that time we called the project PhoneGap and we realized our offering
was playing cat and mouse with the very fast moving dependencies of iOS and
Android. Being reactionary made shipping a fire drill, inevitably drawn out
since we didn't exercise those muscles enough, and ultimately this made our
software a risk for adoption. We didn't want to be a risk for adoption. We
also did not want our volunteer committership killing themselves every time
iOS or Android landed a patch.

Moving to a schedule acted as a forcing function to exercise those muscles,
find our cadence, and only positives to the code and community
resulted. Shipping brought our core together. It meant if we didn't have a
fix for a feature the branch would land in the next release which is only a
month away. This built huge confidence in our team by our community. Our
code become better tested, and more streamlined. A consistent release
cadence not only helped us find more quality in our code, but that
confidence really helped us build our committer and developer community.
The story is hardly unique: Chrome, Ubuntu, Docker, Firefox, and many
others have adopted this model in recent years.

I feel anything that can be considered a better practice for higher quality
code and driving community confidence, and subsequently adoption, really
embodies Apache ideals.

The current process could be largely automated and the vote doesn't
necessarily have to be in the form of an email. I've found these past weeks
the act of voting seems near cultural at Apache so I hope that doesn't
sound crazy! I mean well.

Another issue I am personally unclear on is the wide variety of artifacts
destinations that an Apache project can be shipped today. Maven has some of
these smarts built in but projects like the npm registry do not. Another
area we need to address is the proliferation of various app stores. I'm not
a fan of them, but they happen, and we should have a mechanism for our
projects to deliver to them.


On Fri, Feb 7, 2014 at 3:02 AM, Stephen Connolly <
stephen.alan.connolly@gmail.com> wrote:

> One of the projects I am involved with is the Jenkins project. At Jenkins
> we cut a release of Jenkins every wednesday... assuming the test all pass...
> Not every release is as stable as people might desire (the tests don't
> catch everything), hence there is the LTS release line, but none the less,
> there is a major release every 7 days... and if you look at the usage stats
> (e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg)
> most users actually stick fairly close to the latest release.
>
> I have found that this 7 day release cadence can be really helpful for
> some code bases.
>
> When I started to think about could we follow this model for the Maven
> project as we move towards Maven 4.0, there is one thing that gets in the
> way... namely release votes.
>
> The standard answer is that we could publish snapshots... but those are
> not indented for use by users... and where the cadence can help is that
> these things can be picked up by users.
>
> So what is it that gets in the way with release votes:
>
> * The 72h "soft" requirement for vote duration
>
> * The actions that a PMC member is required to perform before they can
> vote. See http://www.apache.org/dev/release which states:
>
>     > Before voting +1 PMC members are required to download the signed
> source code package, compile it as provided, and test the resulting
> executable on their own platform, along with also verifying that the
> package meets the requirements of the ASF policy on releases.
>
> So how exactly do these things get in the way?
>
> Well as I see it the 72h vote duration isn't necessarily a big deal... we
> need some duration of notice about what is going into the release, there
> will always be people who feel the duration is either too short or two
> long... but with a 7 day cadence and maybe a few hours before the release
> manager closes out the vote and then you wait for the release to finished
> syncing to the mirrors and then the release manager gets a chance to verify
> that the release has synced to at least one mirror... you could easily lose
> half a day's duration in that process... oh look the release is out 3.5
> days after it was cut... and we're cutting another one in 3.5 days... it is
> likely we will not get much meaningful feedback from users in the remaining
> 3.5 days... so essentially you end up with a ping-pong of break... skip...
> fix since if a bleeding edge user finds an issue in 4.0.56 we will have cut
> 4.0.57 by the time they report it to us and the fix ends up in 4.0.58...
> with a shorter vote duration, say 12h, the bleeding edge user reports the
> issue, we fix and the next release is the one they can use.
>
> In the context of a fast cadence, where every committer in the community
> knows there will be a release on wednesday cut from the last revision that
> passed all the tests on the CI system unless there have been no commits
> since the last release that meet that criteria, do we need to wait the full
> 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast
> during those 12h... and if not, well just extend until enough votes are
> cast)
>
> I think this is different use case from my understanding of the concerns
> that drove the 72h vote duration convention, as this would not be 3 PMC
> members who all work for the same company and are in the same location
> conspiring to drive their changes into the release... everything would be
> happening in the open and a 12h window mid-week should allow at least 4h of
> waking time in any TZ.
>
> So the second issue is what a PMC member is required to do before voting...
>
> As a PMC member you are required to
>
> 1. Download the source code package
> 2. Compile it as provided
> 3. Test the resulting executable on your own platform
> 4. Verify that the package meets the requirements of the ASF policy on
> releases
>
> Do we really have to personally do all that *by hand*?
>
> Why can we not have a trusted build server hosted on Apache hardware do
> the download of the source package, compile it as provided and run the
> automated acceptance tests (on a range of platforms), the RAT tooling and
> perhaps verify that the source code package matches what is in source
> control? The trusted build server could then report the results to the
> project mailing list and then the PMC members just need to confirm the
> build server said OK, review the commits between the last time they looked
> at the commits and the tag (which they know matches what is in the source
> bundle) and then vote +1?
>
> The PMC members are supposed to be keeping an eye on the commits anyway,
> so that shouldn't be too onerous, and the release manager could even
> provide a link to the build server confirmation build in the VOTE email.
>
> I would appreciate any people's thoughts on the above.
>
> -Stephen
>
> P.S.
> * Speaking in my personal capacity as a member of the ASF.
> * I am not saying that Maven will move to such a model, or even wants to
> move to such a model... more that I was thinking about the issues that
> might prevent us if we so desired... I know other projects at Apache are
> interested in fast release cadence however, so getting this topic discussed
> in the open is no bad thing IMHO
>
>

Re: How can we support a faster release cadence?

[Stephen Connolly <st...@gmail.com>]

To be clear, I am not sure if Maven wants such a rapid release cycle...
after all we have been supposed to release 3.2.0 since Oct 1st 2013... and
it's still not ready... The other way of looking at it is that if we did
have a 1 release every 7 days we'd probably have 3.2.0 out already!

There are other Apache projects that for sure would like to have the 7 day
cadence, so this is a question that has general relevance at Apache


On 7 February 2014 11:44, Bruno P. Kinoshita <br...@yahoo.com.br>wrote:

> Hi Stephen, @all
>
> I'm not involved in the Maven development, but I enjoy writing plug-ins
> for
> Jenkins.
>
> From a plug-in developer point of view, whenever a developer needs a
> change
> in the Jenkins core, s/he knows that once there is a fix/pull request
> for his issue, it will get released probably within one or two weeks after
> it gets merged into master branch.
>
> So I imagine that developers of plug-ins for Maven could benefit of having
> shorter release cycles too.
>
> I also see lots of bugs being filed in Jenkins JIRA once a release is
> out. Although it is a good practice to use the LTS, I think many users
> install the latest version. This helps to find bugs before they are
> released in the LTS version (sometimes it's hard to write tests for
> all envs + variables).
>
> So my +1 (probably not binding :)
>
> Bruno P. Kinoshita
> http://kinoshita.eti.br
> http://tupilabs.com
>
> >________________________________
> > From: Stephen Connolly <st...@gmail.com>
> >To: dev@community.apache.org
> >Sent: Friday, February 7, 2014 9:14 AM
> >Subject: Re: How can we support a faster release cadence?
> >
> >
> >On 7 February 2014 11:02, Stephen Connolly
> ><st...@gmail.com>wrote:
> >
> >> One of the projects I am involved with is the Jenkins project. At
> Jenkins
> >> we cut a release of Jenkins every wednesday... assuming the test all
> pass...
> >> Not every release is as stable as people might desire (the tests don't
> >> catch everything), hence there is the LTS release line, but none the
> less,
> >> there is a major release every 7 days... and if you look at the usage
> stats
> >> (e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg)
> >> most users actually stick fairly close to the latest release.
> >>
> >> I have found that this 7 day release cadence can be really helpful for
> >> some code bases.
> >>
> >> When I started to think about could we follow this model for the Maven
> >> project as we move towards Maven 4.0, there is one thing that gets in
> the
> >> way... namely release votes.
> >>
> >> The standard answer is that we could publish snapshots... but those are
> >> not indented for use by users... and where the cadence can help is that
> >> these things can be picked up by users.
> >>
> >> So what is it that gets in the way with release votes:
> >>
> >> * The 72h "soft" requirement for vote duration
> >>
> >> * The actions that a PMC member is required to perform before they can
> >> vote. See http://www.apache.org/dev/release which states:
> >>
> >>     > Before voting +1 PMC members are required to download the signed
> >> source code package, compile it as provided, and test the resulting
> >> executable on their own platform, along with also verifying that the
> >> package meets the requirements of the ASF policy on releases.
> >>
> >> So how exactly do these things get in the way?
> >>
> >> Well as I see it the 72h vote duration isn't necessarily a big deal...
> we
> >> need some duration of notice about what is going into the release, there
> >> will always be people who feel the duration is either too short or two
> >> long... but with a 7 day cadence and maybe a few hours before the
> release
> >> manager closes out the vote and then you wait for the release to
> finished
> >> syncing to the mirrors and then the release manager gets a chance to
> verify
> >> that the release has synced to at least one mirror... you could easily
> lose
> >> half a day's duration in that process...
> >>
> >
> >My bad... looking at http://www.apache.org/dev/release I missed
> >
> >> Please ensure that you wait at least 24 hours after uploading a new
> >release before updating the project download page and sending the
> >announcement email(s).
> >
> >So that basically means it could be 4.5 days after the release is cut
> >before it is announced as available to users... wait an average of 12h for
> >a user to download it, add another 12h for them to identify the bug/issue
> >and report it with a test case... That leaves maybe a 1.5 day window for
> >the committers to fix the issue the user has...
> >
> >
> >> oh look the release is out 3.5 days after it was cut... and we're
> cutting
> >> another one in 3.5 days... it is likely we will not get much meaningful
> >> feedback from users in the remaining 3.5 days... so essentially you end
> up
> >> with a ping-pong of break... skip... fix since if a bleeding edge user
> >> finds an issue in 4.0.56 we will have cut 4.0.57 by the time they
> report it
> >> to us and the fix ends up in 4.0.58... with a shorter vote duration, say
> >> 12h, the bleeding edge user reports the issue, we fix and the next
> release
> >> is the one they can use.
> >>
> >
> >The point of a fast cadence is that users know if they have an issue and
> >report it reasonably early enough, they will get the fix in the next
> >release... if we lose 4.5 days between the cutting of the RC and the [ANN]
> >email we can actually harm the community, at least from my experience with
> >the Jenkins project with respect to user engagement.
> >
> >
> >
> >>
> >> In the context of a fast cadence, where every committer in the community
> >> knows there will be a release on wednesday cut from the last revision
> that
> >> passed all the tests on the CI system unless there have been no commits
> >> since the last release that meet that criteria, do we need to wait the
> full
> >> 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get
> cast
> >> during those 12h... and if not, well just extend until enough votes are
> >> cast)
> >>
> >> I think this is different use case from my understanding of the concerns
> >> that drove the 72h vote duration convention, as this would not be 3 PMC
> >> members who all work for the same company and are in the same location
> >> conspiring to drive their changes into the release... everything would
> be
> >> happening in the open and a 12h window mid-week should allow at least
> 4h of
> >> waking time in any TZ.
> >>
> >> So the second issue is what a PMC member is required to do before
> voting...
> >>
> >> As a PMC member you are required to
> >>
> >> 1. Download the source code package
> >> 2. Compile it as provided
> >> 3. Test the resulting executable on your own platform
> >> 4. Verify that the package meets the requirements of the ASF policy on
> >> releases
> >>
> >> Do we really have to personally do all that *by hand*?
> >>
> >> Why can we not have a trusted build server hosted on Apache hardware do
> >> the download of the source package, compile it as provided and run the
> >> automated acceptance tests (on a range of platforms), the RAT tooling
> and
> >> perhaps verify that the source code package matches what is in source
> >> control? The trusted build server could then report the results to the
> >> project mailing list and then the PMC members just need to confirm the
> >> build server said OK, review the commits between the last time they
> looked
> >> at the commits and the tag (which they know matches what is in the
> source
> >> bundle) and then vote +1?
> >>
> >> The PMC members are supposed to be keeping an eye on the commits anyway,
> >> so that shouldn't be too onerous, and the release manager could even
> >> provide a link to the build server confirmation build in the VOTE email.
> >>
> >> I would appreciate any people's thoughts on the above.
> >>
> >> -Stephen
> >>
> >> P.S.
> >> * Speaking in my personal capacity as a member of the ASF.
> >> * I am not saying that Maven will move to such a model, or even wants to
> >> move to such a model... more that I was thinking about the issues that
> >> might prevent us if we so desired... I know other projects at Apache are
> >> interested in fast release cadence however, so getting this topic
> discussed
> >> in the open is no bad thing IMHO
> >>
> >>
> >
> >
> >
>

Re: How can we support a faster release cadence?

["Bruno P. Kinoshita" <br...@yahoo.com.br>]

Hi Stephen, @all

I'm not involved in the Maven development, but I enjoy writing plug-ins for 
Jenkins. 

From a plug-in developer point of view, whenever a developer needs a change 
in the Jenkins core, s/he knows that once there is a fix/pull request 
for his issue, it will get released probably within one or two weeks after 
it gets merged into master branch. 

So I imagine that developers of plug-ins for Maven could benefit of having 
shorter release cycles too.

I also see lots of bugs being filed in Jenkins JIRA once a release is 
out. Although it is a good practice to use the LTS, I think many users 
install the latest version. This helps to find bugs before they are 
released in the LTS version (sometimes it's hard to write tests for 
all envs + variables).

So my +1 (probably not binding :) 

Bruno P. Kinoshita
http://kinoshita.eti.br
http://tupilabs.com

>________________________________
> From: Stephen Connolly <st...@gmail.com>
>To: dev@community.apache.org 
>Sent: Friday, February 7, 2014 9:14 AM
>Subject: Re: How can we support a faster release cadence?
> 
>
>On 7 February 2014 11:02, Stephen Connolly
><st...@gmail.com>wrote:
>
>> One of the projects I am involved with is the Jenkins project. At Jenkins
>> we cut a release of Jenkins every wednesday... assuming the test all pass...
>> Not every release is as stable as people might desire (the tests don't
>> catch everything), hence there is the LTS release line, but none the less,
>> there is a major release every 7 days... and if you look at the usage stats
>> (e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg)
>> most users actually stick fairly close to the latest release.
>>
>> I have found that this 7 day release cadence can be really helpful for
>> some code bases.
>>
>> When I started to think about could we follow this model for the Maven
>> project as we move towards Maven 4.0, there is one thing that gets in the
>> way... namely release votes.
>>
>> The standard answer is that we could publish snapshots... but those are
>> not indented for use by users... and where the cadence can help is that
>> these things can be picked up by users.
>>
>> So what is it that gets in the way with release votes:
>>
>> * The 72h "soft" requirement for vote duration
>>
>> * The actions that a PMC member is required to perform before they can
>> vote. See http://www.apache.org/dev/release which states:
>>
>>     > Before voting +1 PMC members are required to download the signed
>> source code package, compile it as provided, and test the resulting
>> executable on their own platform, along with also verifying that the
>> package meets the requirements of the ASF policy on releases.
>>
>> So how exactly do these things get in the way?
>>
>> Well as I see it the 72h vote duration isn't necessarily a big deal... we
>> need some duration of notice about what is going into the release, there
>> will always be people who feel the duration is either too short or two
>> long... but with a 7 day cadence and maybe a few hours before the release
>> manager closes out the vote and then you wait for the release to finished
>> syncing to the mirrors and then the release manager gets a chance to verify
>> that the release has synced to at least one mirror... you could easily lose
>> half a day's duration in that process...
>>
>
>My bad... looking at http://www.apache.org/dev/release I missed
>
>> Please ensure that you wait at least 24 hours after uploading a new
>release before updating the project download page and sending the
>announcement email(s).
>
>So that basically means it could be 4.5 days after the release is cut
>before it is announced as available to users... wait an average of 12h for
>a user to download it, add another 12h for them to identify the bug/issue
>and report it with a test case... That leaves maybe a 1.5 day window for
>the committers to fix the issue the user has...
>
>
>> oh look the release is out 3.5 days after it was cut... and we're cutting
>> another one in 3.5 days... it is likely we will not get much meaningful
>> feedback from users in the remaining 3.5 days... so essentially you end up
>> with a ping-pong of break... skip... fix since if a bleeding edge user
>> finds an issue in 4.0.56 we will have cut 4.0.57 by the time they report it
>> to us and the fix ends up in 4.0.58... with a shorter vote duration, say
>> 12h, the bleeding edge user reports the issue, we fix and the next release
>> is the one they can use.
>>
>
>The point of a fast cadence is that users know if they have an issue and
>report it reasonably early enough, they will get the fix in the next
>release... if we lose 4.5 days between the cutting of the RC and the [ANN]
>email we can actually harm the community, at least from my experience with
>the Jenkins project with respect to user engagement.
>
>
>
>>
>> In the context of a fast cadence, where every committer in the community
>> knows there will be a release on wednesday cut from the last revision that
>> passed all the tests on the CI system unless there have been no commits
>> since the last release that meet that criteria, do we need to wait the full
>> 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast
>> during those 12h... and if not, well just extend until enough votes are
>> cast)
>>
>> I think this is different use case from my understanding of the concerns
>> that drove the 72h vote duration convention, as this would not be 3 PMC
>> members who all work for the same company and are in the same location
>> conspiring to drive their changes into the release... everything would be
>> happening in the open and a 12h window mid-week should allow at least 4h of
>> waking time in any TZ.
>>
>> So the second issue is what a PMC member is required to do before voting...
>>
>> As a PMC member you are required to
>>
>> 1. Download the source code package
>> 2. Compile it as provided
>> 3. Test the resulting executable on your own platform
>> 4. Verify that the package meets the requirements of the ASF policy on
>> releases
>>
>> Do we really have to personally do all that *by hand*?
>>
>> Why can we not have a trusted build server hosted on Apache hardware do
>> the download of the source package, compile it as provided and run the
>> automated acceptance tests (on a range of platforms), the RAT tooling and
>> perhaps verify that the source code package matches what is in source
>> control? The trusted build server could then report the results to the
>> project mailing list and then the PMC members just need to confirm the
>> build server said OK, review the commits between the last time they looked
>> at the commits and the tag (which they know matches what is in the source
>> bundle) and then vote +1?
>>
>> The PMC members are supposed to be keeping an eye on the commits anyway,
>> so that shouldn't be too onerous, and the release manager could even
>> provide a link to the build server confirmation build in the VOTE email.
>>
>> I would appreciate any people's thoughts on the above.
>>
>> -Stephen
>>
>> P.S.
>> * Speaking in my personal capacity as a member of the ASF.
>> * I am not saying that Maven will move to such a model, or even wants to
>> move to such a model... more that I was thinking about the issues that
>> might prevent us if we so desired... I know other projects at Apache are
>> interested in fast release cadence however, so getting this topic discussed
>> in the open is no bad thing IMHO
>>
>>
>
>
> 

Re: How can we support a faster release cadence?

[Stephen Connolly <st...@gmail.com>]

On 7 February 2014 11:02, Stephen Connolly
<st...@gmail.com>wrote:

> One of the projects I am involved with is the Jenkins project. At Jenkins
> we cut a release of Jenkins every wednesday... assuming the test all pass...
> Not every release is as stable as people might desire (the tests don't
> catch everything), hence there is the LTS release line, but none the less,
> there is a major release every 7 days... and if you look at the usage stats
> (e.g. http://stats.jenkins-ci.org/jenkins-stats/svg/201312-jenkins.svg)
> most users actually stick fairly close to the latest release.
>
> I have found that this 7 day release cadence can be really helpful for
> some code bases.
>
> When I started to think about could we follow this model for the Maven
> project as we move towards Maven 4.0, there is one thing that gets in the
> way... namely release votes.
>
> The standard answer is that we could publish snapshots... but those are
> not indented for use by users... and where the cadence can help is that
> these things can be picked up by users.
>
> So what is it that gets in the way with release votes:
>
> * The 72h "soft" requirement for vote duration
>
> * The actions that a PMC member is required to perform before they can
> vote. See http://www.apache.org/dev/release which states:
>
>     > Before voting +1 PMC members are required to download the signed
> source code package, compile it as provided, and test the resulting
> executable on their own platform, along with also verifying that the
> package meets the requirements of the ASF policy on releases.
>
> So how exactly do these things get in the way?
>
> Well as I see it the 72h vote duration isn't necessarily a big deal... we
> need some duration of notice about what is going into the release, there
> will always be people who feel the duration is either too short or two
> long... but with a 7 day cadence and maybe a few hours before the release
> manager closes out the vote and then you wait for the release to finished
> syncing to the mirrors and then the release manager gets a chance to verify
> that the release has synced to at least one mirror... you could easily lose
> half a day's duration in that process...
>

My bad... looking at http://www.apache.org/dev/release I missed

> Please ensure that you wait at least 24 hours after uploading a new
release before updating the project download page and sending the
announcement email(s).

So that basically means it could be 4.5 days after the release is cut
before it is announced as available to users... wait an average of 12h for
a user to download it, add another 12h for them to identify the bug/issue
and report it with a test case... That leaves maybe a 1.5 day window for
the committers to fix the issue the user has...


> oh look the release is out 3.5 days after it was cut... and we're cutting
> another one in 3.5 days... it is likely we will not get much meaningful
> feedback from users in the remaining 3.5 days... so essentially you end up
> with a ping-pong of break... skip... fix since if a bleeding edge user
> finds an issue in 4.0.56 we will have cut 4.0.57 by the time they report it
> to us and the fix ends up in 4.0.58... with a shorter vote duration, say
> 12h, the bleeding edge user reports the issue, we fix and the next release
> is the one they can use.
>

The point of a fast cadence is that users know if they have an issue and
report it reasonably early enough, they will get the fix in the next
release... if we lose 4.5 days between the cutting of the RC and the [ANN]
email we can actually harm the community, at least from my experience with
the Jenkins project with respect to user engagement.


>
> In the context of a fast cadence, where every committer in the community
> knows there will be a release on wednesday cut from the last revision that
> passed all the tests on the CI system unless there have been no commits
> since the last release that meet that criteria, do we need to wait the full
> 72h for a vote? Would 12h be sufficient (assuming the 3 PMC +1's get cast
> during those 12h... and if not, well just extend until enough votes are
> cast)
>
> I think this is different use case from my understanding of the concerns
> that drove the 72h vote duration convention, as this would not be 3 PMC
> members who all work for the same company and are in the same location
> conspiring to drive their changes into the release... everything would be
> happening in the open and a 12h window mid-week should allow at least 4h of
> waking time in any TZ.
>
> So the second issue is what a PMC member is required to do before voting...
>
> As a PMC member you are required to
>
> 1. Download the source code package
> 2. Compile it as provided
> 3. Test the resulting executable on your own platform
> 4. Verify that the package meets the requirements of the ASF policy on
> releases
>
> Do we really have to personally do all that *by hand*?
>
> Why can we not have a trusted build server hosted on Apache hardware do
> the download of the source package, compile it as provided and run the
> automated acceptance tests (on a range of platforms), the RAT tooling and
> perhaps verify that the source code package matches what is in source
> control? The trusted build server could then report the results to the
> project mailing list and then the PMC members just need to confirm the
> build server said OK, review the commits between the last time they looked
> at the commits and the tag (which they know matches what is in the source
> bundle) and then vote +1?
>
> The PMC members are supposed to be keeping an eye on the commits anyway,
> so that shouldn't be too onerous, and the release manager could even
> provide a link to the build server confirmation build in the VOTE email.
>
> I would appreciate any people's thoughts on the above.
>
> -Stephen
>
> P.S.
> * Speaking in my personal capacity as a member of the ASF.
> * I am not saying that Maven will move to such a model, or even wants to
> move to such a model... more that I was thinking about the issues that
> might prevent us if we so desired... I know other projects at Apache are
> interested in fast release cadence however, so getting this topic discussed
> in the open is no bad thing IMHO
>
>