You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Ben Reser <be...@reser.org> on 2013/08/29 23:47:51 UTC

Improving our release process

Daniel and I had a chat about improving our release process today and I'd like
to make some proposals.

1) Decrease the amount of testing done by the every commit buildbots on the
branches.  This bots should be a quick sanity check, does it build and does it
pass basic tests.

2) Create a buildbot setup that generates a tarball on each branch nightly (if
there were any changes).  It would do this in the same process that a RM would
use (and eventually would become the process for producing the tarball).

3) This tarball would then have various build bots run tests off it.  These
tests would be much more extensive.  In particular it would run cross version
(SERVER_MINOR_VERSION), packing, sharding and other variations of our test
suite that frankly probably don't get as much attention now as they should.

4) Eventually make it so that the RM asks the buildbot to produce the release
tarball, including running the nightly buildbots against it.  RM then downloads
the tarball, compares it to the branch@revision for differences looking for any
unexpected diffs, and then signs it.  It gets put up for everyone to approve.

5) Humans would no longer do much testing.  We'd have buildbots for the major
platforms, which would have been included by our manual testing from the past.
 The build bots would be running far more extensive tests than we ever would be
likely to run on our own.  So the only testing we'd need to do is verifying
that the builds match the source we expect and whatever sanity testing we felt
like we wanted to do.

This should lower the bar dramatically for getting tests/signatures out.
Humans don't need to do this boring and repetitive work.  It should let
everyone spend more time on more important issues than manual testing.

Infra had offered us some more buildbot resources here recently.  This seems
like the logical way to use them.

Re: Improving our release process

Posted by Daniel Shahaf <da...@apache.org>.
On Fri, Aug 30, 2013 at 11:16:33PM +0200, Stefan Sperling wrote:
> On Fri, Aug 30, 2013 at 02:09:20PM -0700, Ben Reser wrote:
> > On 8/30/13 4:01 AM, Bert Huijben wrote:
> > > The openbsd bot is now broken for almost a week, and with a setup to only
> > > test once per 24 hours more bots would be in this state for longer periods.
> > 
> > Agreed, The openbsd bot is a great example of what's not working.
> 
> Well, it's the only bot that actually runs tests like we do
> for a release. The other bots don't run all test combinations.
> 
> I would hope that the proposed release branch build bots would
> be just as thorough as the openbsd bot is.

I'd hope the new bots don't waste their time every (fs, ra) combination
(instead they should just cover each fs and ra backend at least once) and
run "1.7 client, 1.8 server" and "1.7 server, 1.8 client" tests.

Re: Improving our release process

Posted by Ben Reser <be...@reser.org>.
On 8/30/13 2:16 PM, Stefan Sperling wrote:
> Well, it's the only bot that actually runs tests like we do
> for a release. The other bots don't run all test combinations.
>
> I would hope that the proposed release branch build bots would
> be just as thorough as the openbsd bot is.

Right, it's much more complete than any other build bot.  But taking nearly a
day to run one cycle is too long.  I'd like to think even our most complete
tests should be able to be done in 3 hours max, preferably 1-2 hours.

My suggestion for quick sanity checks was with the idea that those can always
be completed in 15-30 minutes.  The point of this is at least partially to
hedge our bets about being able to achieve the above.  If we have fast but
simple tests that finish in less than an hour, then I don't think it would be a
huge deal if the complete tests took 6 hours.

> The recent issue was with serf suddenly requiring a recent scons version
> that isn't part of OpenBSD 5.3. I've fixed that today by updating scons.

Thanks.



Re: Improving our release process

Posted by Stefan Sperling <st...@elego.de>.
On Fri, Aug 30, 2013 at 02:09:20PM -0700, Ben Reser wrote:
> On 8/30/13 4:01 AM, Bert Huijben wrote:
> > The openbsd bot is now broken for almost a week, and with a setup to only
> > test once per 24 hours more bots would be in this state for longer periods.
> 
> Agreed, The openbsd bot is a great example of what's not working.

Well, it's the only bot that actually runs tests like we do
for a release. The other bots don't run all test combinations.

I would hope that the proposed release branch build bots would
be just as thorough as the openbsd bot is.

The recent issue was with serf suddenly requiring a recent scons version
that isn't part of OpenBSD 5.3. I've fixed that today by updating scons.

Re: Improving our release process

Posted by Daniel Shahaf <da...@apache.org>.
On Fri, Aug 30, 2013 at 02:09:20PM -0700, Ben Reser wrote:
> On 8/30/13 4:01 AM, Bert Huijben wrote:
> >On 8/30/13 1:37 AM, Julian Foad wrote:
> > So we only see problems a day later?
> > And can check our fixes the day after.
> > 
> > Maybe we should then also ask every committer to test on Windows and Unix
> > before every commit?
> > (Or maybe ask them to setup their own bots, because we can't provide them
> > the current test coverage)
> > 
> > Why reduce the testing on the existing buildbots?
> 
> You realize I'm talking about the release branch build bots.  I don't think any
> particular change is needed for trunk build bots.
> 

Usually the stable branches get code changes only when svn-role commits
them, which is in batch from 4am to 4:01am, so the question isn't as much
"daily or post-commit" as it is "Test tonight's batch in aggregate (aka just
test the new HEAD), or test each of tonight's commits separately".

Daniel

> As it is right now the branch build bots are generally broken anyway because
> they live on the same host machines as trunk and trunk marches on and requires
> dependencies that the release branches can't use.
> 
> So even a more basic set of tests on every commit would be an improvement.  I
> also don't think we need as much effort on every commit in the release branches
> because they are review then commit and tend to move much more slowly.
> 
> > We should just add 'fast bots' if we think the current bots are too slow. I
> > don't think new hardware was added at the primary bots for the last 2-3
> > years, so just spending some money there should improve the build times
> > without reducing the test coverage given within a few hours.
> 
> Sure if we can find resources to do more testing fast I'm not objecting to
> that.  I'm just trying to prioritize resources where they are needed most.
> 
> I didn't really mention it but it's critical that the release branches have
> build bots that are stable.  They can't be shared with trunk if we're going to
> use OS packages (and we probably should).

Re: Improving our release process

Posted by Greg Stein <gs...@gmail.com>.
On Fri, Aug 30, 2013 at 4:09 PM, Ben Reser <be...@reser.org> wrote:
>...
> As it is right now the branch build bots are generally broken anyway because
> they live on the same host machines as trunk and trunk marches on and requires
> dependencies that the release branches can't use.

Switch to fully-static builds, and make available different .a library
versions for each of the buildbots running on a box. Should work just
fine.

Cheers,
-g

Re: Improving our release process

Posted by Ben Reser <be...@reser.org>.
On 8/30/13 4:01 AM, Bert Huijben wrote:
>On 8/30/13 1:37 AM, Julian Foad wrote:
> So we only see problems a day later?
> And can check our fixes the day after.
> 
> Maybe we should then also ask every committer to test on Windows and Unix
> before every commit?
> (Or maybe ask them to setup their own bots, because we can't provide them
> the current test coverage)
> 
> Why reduce the testing on the existing buildbots?

You realize I'm talking about the release branch build bots.  I don't think any
particular change is needed for trunk build bots.

As it is right now the branch build bots are generally broken anyway because
they live on the same host machines as trunk and trunk marches on and requires
dependencies that the release branches can't use.

So even a more basic set of tests on every commit would be an improvement.  I
also don't think we need as much effort on every commit in the release branches
because they are review then commit and tend to move much more slowly.

> We should just add 'fast bots' if we think the current bots are too slow. I
> don't think new hardware was added at the primary bots for the last 2-3
> years, so just spending some money there should improve the build times
> without reducing the test coverage given within a few hours.

Sure if we can find resources to do more testing fast I'm not objecting to
that.  I'm just trying to prioritize resources where they are needed most.

I didn't really mention it but it's critical that the release branches have
build bots that are stable.  They can't be shared with trunk if we're going to
use OS packages (and we probably should).

> The openbsd bot is now broken for almost a week, and with a setup to only
> test once per 24 hours more bots would be in this state for longer periods.

Agreed, The openbsd bot is a great example of what's not working.

> Didn't we already have one?
> 
> See:
> http://ci.apache.org/builders/svn-trunk-nightly/builds/875

Yep, but only for trunk.  I'm talking about release branch nightlies.

> I don't see what it would help to run the bots run on tarballs vs running it
> on specific revisions.

Validation of our libtool/autoconf/swig/etc... version choices.  The current
build bot process runs autogen.sh.  Our current process only tests the checkout
from SVN.  However, almost none of our users or packagers use that.

A great example of an error that has persisted for a long time is the bindings
breakage on OS X in our tarballs.  SWIG is run by the RM on Linux, which often
turns triggers some APIs being enabled by the CPP.  Then when you go to build
the bindings on OS X they break.  An average user is going to think our
bindings are broken on OS X if they're building them.  Rather they just need to
know the magic incantation to throw away the already produced C files and have
their own copy of SWIG installed.  (Which reminds me I need to get back to
fixing that).

That's at least part of the reason for us doing manual testing on the tarballs.
 Otherwise, we could just tag, test, generate tarballs and then sign the
tarballs with nothing other than a simple comparison of the source files.

Testing tarballs on the release branches does avoid more diverse autogen.sh
dependencies, but I don't think our build bots really have that diverse of
dependencies in this case anyway.

> It will certainly cost a lot of time to rewrite the scripts, while the
> buildbot can just handle this for tags/branches/revisions.

I realize this isn't entirely cost free.  It's not going to happen overnight.
But it's something we should strive for.

I also don't think it'll take much for the build bots to use tarballs.  Just
have to replace the checkout script with a tarball retrieving/expanding script
and then skip autogen.sh in the existing buld scripts.

> I try to build from tags/revisions whenever possible, as this allows me to
> always get the exact sourcecode for files back via the SCM information
> stored in the debug information without having to keep all versions of all
> tarballs.
> 
> 
> I trust that our tags and revisions in Subversion don't change. Don't you?

I do the same.  I trust our tags and revisions.  That wasn't the point.

>> One more thing we'd need to to before signing, of course: verify that the
>> buildbot test runs have completed to our satisfaction (right tarball,
>> sufficient platforms, etc.).

Right, this was assumed but I should have mentioned it in my original email.

> The manual testing also adds coverage outside the scripted tests. My scripts
> make my tests run almost identical as on the buildbot, but over the last 5
> years I found more than a few release problems in the difference between
> that 'almost identical' and the buildbots itself.

As I noted in my original email, nobody is stopping interested parties from
running additional manual tests if they want to.  I just want to encourage
people to test/sign rather than assuming that someone else will do it because
they aren't excited about spending the time to do the testing.

There will always be things that are missed by the build bots and even manual
testing becuase they are random failures.  However, I'd guess that we'll
probably see more of them if we started running extensive tests on a nightly
basis because the test suites would get run a lot more times than even our
manual testing is showing.


RE: Improving our release process

Posted by Bert Huijben <be...@qqmail.nl>.

> -----Original Message-----
> From: Julian Foad [mailto:julianfoad@btopenworld.com]
> Sent: vrijdag 30 augustus 2013 10:38
> To: Ben Reser
> Cc: Subversion Development
> Subject: Re: Improving our release process
> 
> Ben Reser wrote:
> > 1) Decrease the amount of testing done by the every commit buildbots on
> the
> > branches.  This bots should be a quick sanity check, does it build and
does it
> > pass basic tests.

So we only see problems a day later?
And can check our fixes the day after.

Maybe we should then also ask every committer to test on Windows and Unix
before every commit?
(Or maybe ask them to setup their own bots, because we can't provide them
the current test coverage)

Why reduce the testing on the existing buildbots?

We should just add 'fast bots' if we think the current bots are too slow. I
don't think new hardware was added at the primary bots for the last 2-3
years, so just spending some money there should improve the build times
without reducing the test coverage given within a few hours.

The openbsd bot is now broken for almost a week, and with a setup to only
test once per 24 hours more bots would be in this state for longer periods.

> > 2) Create a buildbot setup that generates a tarball on each branch
nightly (if
> > there were any changes).  It would do this in the same process that a RM
> would
> > use (and eventually would become the process for producing the tarball).

Didn't we already have one?

See:
http://ci.apache.org/builders/svn-trunk-nightly/builds/875


> >
> > 3) This tarball would then have various build bots run tests off it. 
These
> > tests would be much more extensive.  In particular it would run cross
> version
> > (SERVER_MINOR_VERSION), packing, sharding and other variations of our
> test
> > suite that frankly probably don't get as much attention now as they
should.

I don't see what it would help to run the bots run on tarballs vs running it
on specific revisions.

It will certainly cost a lot of time to rewrite the scripts, while the
buildbot can just handle this for tags/branches/revisions.

I try to build from tags/revisions whenever possible, as this allows me to
always get the exact sourcecode for files back via the SCM information
stored in the debug information without having to keep all versions of all
tarballs.


I trust that our tags and revisions in Subversion don't change. Don't you?

> >
> > 4) Eventually make it so that the RM asks the buildbot to produce the
> release
> > tarball, including running the nightly buildbots against it.  RM then
> downloads
> > the tarball, compares it to the branch@revision for differences looking
for
> any
> > unexpected diffs, and then signs it.  It gets put up for everyone to
approve.
> >
> > 5) Humans would no longer do much testing.  We'd have buildbots for the
> major
> > platforms, which would have been included by our manual testing from the
> past.
> > The build bots would be running far more extensive tests than we ever
> would be
> > likely to run on our own.  So the only testing we'd need to do is
verifying
> > that the builds match the source we expect and whatever sanity testing
we
> felt
> > like we wanted to do.
> 
> One more thing we'd need to to before signing, of course: verify that the
> buildbot test runs have completed to our satisfaction (right tarball,
sufficient
> platforms, etc.).

The manual testing also adds coverage outside the scripted tests. My scripts
make my tests run almost identical as on the buildbot, but over the last 5
years I found more than a few release problems in the difference between
that 'almost identical' and the buildbots itself.

> > This should lower the bar dramatically for getting tests/signatures out.
> > Humans don't need to do this boring and repetitive work.  It should let
> > everyone spend more time on more important issues than manual testing.
> >
> > Infra had offered us some more buildbot resources here recently.  This
> seems
> > like the logical way to use them.
> 
> +1 to all you said.
> 
> - Julian


Re: Improving our release process

Posted by Julian Foad <ju...@btopenworld.com>.
Ben Reser wrote:
> 1) Decrease the amount of testing done by the every commit buildbots on the
> branches.  This bots should be a quick sanity check, does it build and does it
> pass basic tests.
> 
> 2) Create a buildbot setup that generates a tarball on each branch nightly (if
> there were any changes).  It would do this in the same process that a RM would
> use (and eventually would become the process for producing the tarball).
> 
> 3) This tarball would then have various build bots run tests off it.  These
> tests would be much more extensive.  In particular it would run cross version
> (SERVER_MINOR_VERSION), packing, sharding and other variations of our test
> suite that frankly probably don't get as much attention now as they should.
> 
> 4) Eventually make it so that the RM asks the buildbot to produce the release
> tarball, including running the nightly buildbots against it.  RM then downloads
> the tarball, compares it to the branch@revision for differences looking for any
> unexpected diffs, and then signs it.  It gets put up for everyone to approve.
> 
> 5) Humans would no longer do much testing.  We'd have buildbots for the major
> platforms, which would have been included by our manual testing from the past.
> The build bots would be running far more extensive tests than we ever would be
> likely to run on our own.  So the only testing we'd need to do is verifying
> that the builds match the source we expect and whatever sanity testing we felt
> like we wanted to do.

One more thing we'd need to to before signing, of course: verify that the buildbot test runs have completed to our satisfaction (right tarball, sufficient platforms, etc.).

> This should lower the bar dramatically for getting tests/signatures out.
> Humans don't need to do this boring and repetitive work.  It should let
> everyone spend more time on more important issues than manual testing.
> 
> Infra had offered us some more buildbot resources here recently.  This seems
> like the logical way to use them.

+1 to all you said.

- Julian