release engineering (was: Re: 1.6.0-rc4 up for signing/testing)

[Stefan Sperling <st...@elego.de>]

[Cc'ing dev@ to gather comments]

On Wed, Mar 18, 2009 at 09:38:25AM -0500, Hyrum K. Wright wrote:
>
> On Mar 18, 2009, at 9:01 AM, Stefan Sperling wrote:
> > And only 3 months delay so far :)
>
> Sure, but it's a lot sooner than 1.5 was, so the 2nd derivative is
> negative. :)
>
>> This may be interesting to you, albeit not quite as applicable to
>> subversion as I'd like it to be:
>> http://www.openbsd.org/papers/asiabsdcon2009-release_engineering/
>
> Ooooo, I like it.  Thanks!

So, could some of this be applied to Subversion?

The essence I think is this slide ("our release cycle is simply
a fine-tuning period"):
http://www.openbsd.org/papers/asiabsdcon2009-release_engineering/mgp00017.html

Because many advanced OpenBSD users run regularly provided snapshots
and update their systems routinely every so often, a lot of
regressions and bugs in new features are caught quickly, often long
before release time.

A key point here is that making snapshots available makes it easy
for people to try out the current state of things. They don't have
to go through all the hassle of checking out trunk, compiling it,
and so on. We know that not many of our own users (especially Windows
users) don't want to bother with that. 

I don't think getting an infrastructure that automatically compiles
Subversion snapshots for download would be incredibly difficult.
It's more of a question of who has the time and will to set it up.
It could probably be integrated with the buildbots, since they are
already compiling and testing binaries anyway. Just add packaging.
Then we'd have regular builds known to pass the regression test suite
for some versions of at least Windows, Mac, and some Linux distros.

The hard problem would be getting advanced Subversion users run
these snapshots in production, instead of regular releases.
This requires trust from the user side ("I can trust the Subversion devs
not to break the snapshots") and thus we developers would need to be
careful and willing to build up enough trust and then not to break it.

People running snaps would then be even more tightly integrated during
the release cycle and would be more readily able to do testing because
upgrading to the latest snap has become routine anyway and does not
take much time.

Whereas currently, we don't get any significant amount of testing by the
user community even closely before release. At least not as much as we'd
like to get. Most problems are reported after a "dot oh" release.
All we know at the time of release is that the test suite passes.
But we don't know if the code survives real-world usage.

I'd say that if we got something like this going, it would do its part
in making the 6 months release cycle dream we have been dreaming become
reality.

This would need not in any way affect the Subversion project's practice
of only releasing source code for official releases.

Stefan

Re: release engineering (was: Re: 1.6.0-rc4 up for signing/testing)

[km...@rockwellcollins.com]

"Hyrum K. Wright" <hy...@mail.utexas.edu> wrote on 03/18/2009 
11:37:39 AM:
> On Mar 18, 2009, at 11:07 AM, Stefan Sperling wrote:
> 
> > [Cc'ing dev@ to gather comments]
> >
> > On Wed, Mar 18, 2009 at 09:38:25AM -0500, Hyrum K. Wright wrote:
> >>
> >> On Mar 18, 2009, at 9:01 AM, Stefan Sperling wrote:
> >>> And only 3 months delay so far :)
> >>
> >> Sure, but it's a lot sooner than 1.5 was, so the 2nd derivative is
> >> negative. :)
> >>
> >>> This may be interesting to you, albeit not quite as applicable to
> >>> subversion as I'd like it to be:
> >>> http://www.openbsd.org/papers/asiabsdcon2009-release_engineering/
> >>
> >> Ooooo, I like it.  Thanks!
> >
> > So, could some of this be applied to Subversion?
> >
> > The essence I think is this slide ("our release cycle is simply
> > a fine-tuning period"):
> > 
http://www.openbsd.org/papers/asiabsdcon2009-release_engineering/mgp00017.html
> >
> > Because many advanced OpenBSD users run regularly provided snapshots
> > and update their systems routinely every so often, a lot of
> > regressions and bugs in new features are caught quickly, often long
> > before release time.
> >
> > A key point here is that making snapshots available makes it easy
> > for people to try out the current state of things. They don't have
> > to go through all the hassle of checking out trunk, compiling it,
> > and so on. We know that not many of our own users (especially Windows
> > users) don't want to bother with that.
> >
> > I don't think getting an infrastructure that automatically compiles
> > Subversion snapshots for download would be incredibly difficult.
> > It's more of a question of who has the time and will to set it up.
> > It could probably be integrated with the buildbots, since they are
> > already compiling and testing binaries anyway. Just add packaging.
> > Then we'd have regular builds known to pass the regression test suite
> > for some versions of at least Windows, Mac, and some Linux distros.
> 
> When we were going through the 1.5 release cycle, I wrote a script to 
> post nighly tarballs on orac.  See http://svn.collab.
> net/repos/svn/trunk/tools/dist/nightly.sh

The biggest problem for 'normal' users using these (or other) snapshots
is the potential working copy incompatibilities between versions.  I've
been caught numerous times saying "oh crap", after I mistakenly updated
a large working copy with a dev version of subversion.

Maybe the wc-ng stuff can help by making things more forwards and
backwards compatible, but I doubt that is even on the list of things
to address.

Don't get me wrong, I think nightly builds are a great idea, just
we can't expect people to consistently use them unless working
copy compatibility is enhanced.

`Kevin R.

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1349472

Re: release engineering (was: Re: 1.6.0-rc4 up for signing/testing)

["Hyrum K. Wright" <hy...@mail.utexas.edu>]

On Mar 18, 2009, at 11:07 AM, Stefan Sperling wrote:

> [Cc'ing dev@ to gather comments]
>
> On Wed, Mar 18, 2009 at 09:38:25AM -0500, Hyrum K. Wright wrote:
>>
>> On Mar 18, 2009, at 9:01 AM, Stefan Sperling wrote:
>>> And only 3 months delay so far :)
>>
>> Sure, but it's a lot sooner than 1.5 was, so the 2nd derivative is
>> negative. :)
>>
>>> This may be interesting to you, albeit not quite as applicable to
>>> subversion as I'd like it to be:
>>> http://www.openbsd.org/papers/asiabsdcon2009-release_engineering/
>>
>> Ooooo, I like it.  Thanks!
>
> So, could some of this be applied to Subversion?
>
> The essence I think is this slide ("our release cycle is simply
> a fine-tuning period"):
> http://www.openbsd.org/papers/asiabsdcon2009-release_engineering/mgp00017.html
>
> Because many advanced OpenBSD users run regularly provided snapshots
> and update their systems routinely every so often, a lot of
> regressions and bugs in new features are caught quickly, often long
> before release time.
>
> A key point here is that making snapshots available makes it easy
> for people to try out the current state of things. They don't have
> to go through all the hassle of checking out trunk, compiling it,
> and so on. We know that not many of our own users (especially Windows
> users) don't want to bother with that.
>
> I don't think getting an infrastructure that automatically compiles
> Subversion snapshots for download would be incredibly difficult.
> It's more of a question of who has the time and will to set it up.
> It could probably be integrated with the buildbots, since they are
> already compiling and testing binaries anyway. Just add packaging.
> Then we'd have regular builds known to pass the regression test suite
> for some versions of at least Windows, Mac, and some Linux distros.


When we were going through the 1.5 release cycle, I wrote a script to  
post nighly tarballs on orac.  See http://svn.collab.net/repos/svn/trunk/tools/dist/nightly.sh

-Hyrum

------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1349388

Re: release engineering (was: Re: 1.6.0-rc4 up for signing/testing)

[Peter Samuelson <pe...@p12n.org>]

[Stefan Sperling]
> I don't think getting an infrastructure that automatically compiles
> Subversion snapshots for download would be incredibly difficult.
[...]
> This would need not in any way affect the Subversion project's
> practice of only releasing source code for official releases.

It would, however, be exactly the opposite of current policy, where we
strongly discourage people from making it easy to install any of our
code that is not based on an official, blessed tarball.

The strongly worded statement Hyrum always includes in his pre-release
announcements suggests to me that the current policy is based on
strongly held opinions.

(Note, I am usually on the side of 'release early', not the side of
'protect the quality of the brand'.)
-- 
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/