BuildBots

[Ben Reser <be...@reser.org>]

We need our buildbots to work for branches.  I propose the following changes,
in decreasing order of priority.

1) If something like bindings is broken on a build bot for branches then
disable the test on that buildbot.  It is far better to disable bindings tests
than it is to continue to allow the build bot to fail and thus encourage us to
ignore the build bot entirely.

2) We should find a way to keep functioning build slaves for our branches that
support the same tests as we run for trunk.  If that means providing parallel
installations of dependencies on the same machine or using separate machines
then we should do that.

3) The build bot should not use the same slave for builds of branch and trunk
changes.  Right now the waterfall view does not give you a good view of the
state of our branches/trunk.  If I break 1.7.x the bot will show red until the
next build which might be on trunk succeeds.  That's not useful.  I can
understand sometimes running a test on a branch that's not normally tested
(like I did with the 1.7.x-diff-translate branch) and using the 1.7.x slave.
But we really ought to have slaves for each release branch we're using.

Obviously 2 and 3 are longer term propositions.  But #1 is something we should
fix immediately.  To that end I have conducted an audit of what's broken on
which build bots and which release branches.  My findings are below.

1.8.x:
  svn-x64-ubuntu-gcc: works
  svn-x64-centos-gcc: works
  bb-openbsd: doesn't build 1.8.x
  svn-windows-local: JavaHL fails.
  svn-windows-ra: Perl bindings fails.

1.7.x:
  svn-x64-ubuntu-gcc: Ruby bindings fail (build not just tests)
  svn-x64-centos-gcc: works
  bb-openbsd: doesn't build 1.8.x
  svn-windows-local: JavaHL fails.
  svn-windows-ra:  Build fails (can't determine why bot is down right now).

We should disable the swig-rb tests on 1.7.x on svn-x864-ubuntu-gcc since the
problem there is that the version of Ruby on that host is too new for 1.7.x.

Bert said the JavaHL tests on svn-windows-local are known due to some sort of
DLL PATH problem.  We should fix or disable these.

I recall Bert having said in the past that SWIG-PL doesn't build on Windows
properly, we should disable that test for swig-windows-ra.

The build bots are down for Windows right now due to a network issue.  So I
can't run a specific test to determine what's wrong with svn-windows-ra and
1.7.x and the logs for the old builds have been removed already.

We have two ways to handle turning tests off by branch.  First we can pass the
branch into the scripts that it's running for and then turn things off.  The
only problem with this is that svncheck.sh for the *nix buildbots already takes
the list of tests to run.  So we'll need to change the scripts to accept this
in parallel with the master changes to pass it.  We could alternatively detect
the branch from the working copy.  But I'm not sure how easy this will be to do.

Or we could just skip all of the above and get the build bots setup with
separate slaves for each branch and then they can have their own independent
scripts.

I'd start on some of the above steps but I'm also not clear on how these
scripts get updated on the slaves.  Does someone have to do that manually, are
they automatically updated?  If they are manually updated are the current
versions in SVN?  For that matter is there someplace where we document exactly
who is responsible for which slaves?

Re: AW: BuildBots

[Ben Reser <be...@reser.org>]

On 2/16/14, 1:22 AM, Branko Čibej wrote:
> On 16.02.2014 10:10, Markus Schaber wrote:
>> Hi, Ben,
>>
>> Von: Ben Reser [ben@reser.org]
>>
>>> 1) If something like bindings is broken on a build bot for branches then
>>> disable the test on that buildbot.  It is far better to disable bindings tests
>>> than it is to continue to allow the build bot to fail and thus encourage us to
>>> ignore the build bot entirely.
>> Is it possible to separate the bindings into a separate build? Thus, we can
>> easily see that the SVN build itsself works, and only the bindings fail?
> 
> We can already see this.

Well yes and no.  If the bindings are being tested out of the same script that
runs the other tests then you can only see that the bindings failed as long as
the logs are still available.  Given that branches don't get built that often
the logs for the most recent branch build can easily be long gone.

Splitting them doesn't really help them towards my purposes here.  If the bot
or notification email says the build fails and doesn't say why someone still
has to look at why.  Once you're looking at why and see failures you know about
you'll eventually start ignoring the bot.

My goal is to make it so every bot failure is considered something that needs
to be looked into.  Release branches even more so than trunk.

Re: AW: BuildBots

[Branko Čibej <br...@wandisco.com>]

On 16.02.2014 10:10, Markus Schaber wrote:
> Hi, Ben,
>
> Von: Ben Reser [ben@reser.org]
>
>> 1) If something like bindings is broken on a build bot for branches then
>> disable the test on that buildbot.  It is far better to disable bindings tests
>> than it is to continue to allow the build bot to fail and thus encourage us to
>> ignore the build bot entirely.
> Is it possible to separate the bindings into a separate build? Thus, we can
> easily see that the SVN build itsself works, and only the bindings fail?

We can already see this.

-- Brane

-- 
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com

AW: BuildBots

[Markus Schaber <m....@codesys.com>]

Hi, Ben,

Von: Ben Reser [ben@reser.org]

> 1) If something like bindings is broken on a build bot for branches then
> disable the test on that buildbot.  It is far better to disable bindings tests
> than it is to continue to allow the build bot to fail and thus encourage us to
> ignore the build bot entirely.

Is it possible to separate the bindings into a separate build? Thus, we can
easily see that the SVN build itsself works, and only the bindings fail?



Best regards

Markus Schaber

(This email was sent from a mobile device...)

CODESYS® a trademark of 3S-Smart Software Solutions GmbH

Inspiring Automation Solutions
________________________________
3S-Smart Software Solutions GmbH
Dipl.-Inf. Markus Schaber | Product Development Core Technology
Memminger Str. 151 | 87439 Kempten | Germany
Tel. +49-831-54031-979 | Fax +49-831-54031-50

E-Mail: m.schaber@codesys.com | Web: codesys.com
CODESYS internet forum: forum.codesys.com

Managing Directors: Dipl.Inf. Dieter Hess, Dipl.Inf. Manfred Werner | Trade register: Kempten HRB 6186 | Tax ID No.: DE 167014915

________________________________________

Re: BuildBots

[Hyrum K Wright <hy...@hyrumwright.org>]

On Tue, Feb 18, 2014 at 12:51 PM, Ben Reser <be...@reser.org> wrote:

> On 2/17/14, 12:38 PM, Hyrum K Wright wrote:
> > I know that on the svn-x64-ubuntu-gcc slave (currently residing in my
> > basement), the scripts are updated manually.  I may even have local
> edits that
> > aren't yet in the repository.
>
> I suspected as much.  Can I possibly get access to this machine to work on
> resolving this?
>

Sure, but it might take me a day or two.  I'll ping you in IRC.


>
> > On Fri, Feb 14, 2014 at 6:49 PM, Ben Reser <ben@reser.org
> > <ma...@reser.org>> wrote:
> >     versions in SVN?  For that matter is there someplace where we
> document exactly
> >     who is responsible for which slaves?
> >
> >
> > +1 to documenting who owns what.
>
> It is documented on the builders page, I just didn't know this.
>
> E.G.
> https://ci.apache.org/builders/svn-x64-centos-gcc
>

Re: BuildBots

[Ben Reser <be...@reser.org>]

On 2/17/14, 12:38 PM, Hyrum K Wright wrote:
> I know that on the svn-x64-ubuntu-gcc slave (currently residing in my
> basement), the scripts are updated manually.  I may even have local edits that
> aren't yet in the repository.

I suspected as much.  Can I possibly get access to this machine to work on
resolving this?

> On Fri, Feb 14, 2014 at 6:49 PM, Ben Reser <ben@reser.org
> <ma...@reser.org>> wrote:
>     versions in SVN?  For that matter is there someplace where we document exactly
>     who is responsible for which slaves?
> 
> 
> +1 to documenting who owns what. 

It is documented on the builders page, I just didn't know this.

E.G.
https://ci.apache.org/builders/svn-x64-centos-gcc

Re: BuildBots

[Hyrum K Wright <hy...@hyrumwright.org>]

On Fri, Feb 14, 2014 at 6:49 PM, Ben Reser <be...@reser.org> wrote:

> We need our buildbots to work for branches.  I propose the following
> changes,
> in decreasing order of priority.
>
> 1) If something like bindings is broken on a build bot for branches then
> disable the test on that buildbot.  It is far better to disable bindings
> tests
> than it is to continue to allow the build bot to fail and thus encourage
> us to
> ignore the build bot entirely.
>
> 2) We should find a way to keep functioning build slaves for our branches
> that
> support the same tests as we run for trunk.  If that means providing
> parallel
> installations of dependencies on the same machine or using separate
> machines
> then we should do that.
>
> 3) The build bot should not use the same slave for builds of branch and
> trunk
> changes.  Right now the waterfall view does not give you a good view of the
> state of our branches/trunk.  If I break 1.7.x the bot will show red until
> the
> next build which might be on trunk succeeds.  That's not useful.  I can
> understand sometimes running a test on a branch that's not normally tested
> (like I did with the 1.7.x-diff-translate branch) and using the 1.7.x
> slave.
> But we really ought to have slaves for each release branch we're using.
>
> Obviously 2 and 3 are longer term propositions.  But #1 is something we
> should
> fix immediately.  To that end I have conducted an audit of what's broken on
> which build bots and which release branches.  My findings are below.
>
> 1.8.x:
>   svn-x64-ubuntu-gcc: works
>   svn-x64-centos-gcc: works
>   bb-openbsd: doesn't build 1.8.x
>   svn-windows-local: JavaHL fails.
>   svn-windows-ra: Perl bindings fails.
>
> 1.7.x:
>   svn-x64-ubuntu-gcc: Ruby bindings fail (build not just tests)
>   svn-x64-centos-gcc: works
>   bb-openbsd: doesn't build 1.8.x
>   svn-windows-local: JavaHL fails.
>   svn-windows-ra:  Build fails (can't determine why bot is down right now).
>
> We should disable the swig-rb tests on 1.7.x on svn-x864-ubuntu-gcc since
> the
> problem there is that the version of Ruby on that host is too new for
> 1.7.x.
>
> Bert said the JavaHL tests on svn-windows-local are known due to some sort
> of
> DLL PATH problem.  We should fix or disable these.
>
> I recall Bert having said in the past that SWIG-PL doesn't build on Windows
> properly, we should disable that test for swig-windows-ra.
>
> The build bots are down for Windows right now due to a network issue.  So I
> can't run a specific test to determine what's wrong with svn-windows-ra and
> 1.7.x and the logs for the old builds have been removed already.
>
> We have two ways to handle turning tests off by branch.  First we can pass
> the
> branch into the scripts that it's running for and then turn things off.
>  The
> only problem with this is that svncheck.sh for the *nix buildbots already
> takes
> the list of tests to run.  So we'll need to change the scripts to accept
> this
> in parallel with the master changes to pass it.  We could alternatively
> detect
> the branch from the working copy.  But I'm not sure how easy this will be
> to do.
>
> Or we could just skip all of the above and get the build bots setup with
> separate slaves for each branch and then they can have their own
> independent
> scripts.
>
> I'd start on some of the above steps but I'm also not clear on how these
> scripts get updated on the slaves.  Does someone have to do that manually,
> are
> they automatically updated?  If they are manually updated are the current
>

I know that on the svn-x64-ubuntu-gcc slave (currently residing in my
basement), the scripts are updated manually.  I may even have local edits
that aren't yet in the repository.


> versions in SVN?  For that matter is there someplace where we document
> exactly
> who is responsible for which slaves?
>

+1 to documenting who owns what.