You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Justin Mason <jm...@jmason.org> on 2004/12/17 19:14:16 UTC

Re: buildbot failure in [...]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Malte S. Stretz writes:
> On Friday 17 December 2004 11:36 CET buildbot@bugzilla.spamassassin.org 
> wrote:
> > The Buildbot has detected a new failure of trunk-debian-stable.
> >
> > Buildbot URL: http://bugzilla.spamassassin.org:8010/
> >
> > Build Reason: changes
> > Build Source Stamp: 112
> > Blamelist: quinlan
> >
> > BUILD FAILED: failed svn
> 
> Those messages are getting a bit annoying, is there any way to filter any 
> builtbot message which contains "BUILD FAILED: failed svn" on the server?

no!

(a) they really are failures.  in this case the svn server seems to have
died, which is good to know ;)   The whole point of this is to get
notification of failures.

(b) however the -parker- and -sidney- ones *are* getting annoying. ;)  I
suggest we turn off those slaves until we can figure out how to get
buildbot to work with dynamic-IP slaves...

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFBwyH4MJF5cimLx9ARAuMbAJ9SSnez7MSgQtUsq9JlKFnP6t8EEACfYqZo
Cnd/J6zOu6Gqe6h+HHXvKQE=
=X3Rp
-----END PGP SIGNATURE-----


Re: buildbot failure in [...]

Posted by Duncan Findlay <du...@debian.org>.
On Fri, Dec 17, 2004 at 04:00:52PM -0600, Michael Parker wrote:
> The mass svn failure this morning was due to the svn server falling
> over (ie kernel panic) and svn flat out wasn't available.
> 
> The ASF svn installation seems to be having some growing pains, you
> might have noticed revision numbers jumping in orders of magnitudes,
> there are now a good number of projects in the repository.

Ironically, this sort of thing is one of the reasons why we left SF
:-)

-- 
Duncan Findlay

Re: buildbot failure in [...]

Posted by Michael Parker <pa...@pobox.com>.
Yuck, I don't know if --keepalive=300 is going to work.  I added it
and now they disconnect after a few minutes of inactivity.  I'm going
to shut my slaves off for now, I do enough local testing as it is, so
we'll just lose the FreeBSD machine.

Michael

Re: buildbot failure in [...]

Posted by Michael Parker <pa...@pobox.com>.
The mass svn failure this morning was due to the svn server falling
over (ie kernel panic) and svn flat out wasn't available.

The ASF svn installation seems to be having some growing pains, you
might have noticed revision numbers jumping in orders of magnitudes,
there are now a good number of projects in the repository.

I imagine we'll have some growing pains, but luckily several of the
core SVN developers are also ASF members with an interest in making
sure it is all working correctly.

Michael

Re: buildbot failure in [...]

Posted by Sidney Markowitz <si...@sidney.com>.
Malte S. Stretz wrote:
> Does anybody know what exactly goes wrong?  Maybe it could work if we use 
> port forwarding or stunnel or something to route the traffic to the dynamic 
> clients over some server with a static IP?

Here's my last svn failed log. It was on the native machine, and I just 
discovered that the two slaves on the VMWare virtual machine have been 
not responding for a couple of days, so it cannot be a matter of 
simultaneous access on the same machine. I'm going to try to restart them.

Could the svn server be sensitive to too many clients hitting the same 
repository at the same time? Perhaps it would help to introduce a delay 
between triggering one slave and the next, or if that is not possible 
adding a sleep of a random time on the slaves before the svn update.

  -- sidney

The log:


starting svn operation
command '['svn', 'update', '--revision', '122631']' in dir 
/b/home/buildbot/slaves/sidney-fedora3/trunk-sidney-fedora3/build 
(timeout 1200 secs)
svn: PROPFIND request failed on '/repos/asf/spamassassin/trunk'
svn: PROPFIND of '/repos/asf/spamassassin/trunk': Could not read status 
line: connection was closed by server. (http://svn.apache.org)
update failed, clobbering and trying again
command '['rm', '-rf', 
'/b/home/buildbot/slaves/sidney-fedora3/trunk-sidney-fedora3/build']' in 
dir /b/home/buildbot/slaves/sidney-fedora3/trunk-sidney-fedora3 (timeout 
1200 secs)
now retrying VC operation
command '['svn', 'checkout', '--revision', '122631', 
'http://svn.apache.org/repos/asf/spamassassin/trunk', 'build']' in dir 
/b/home/buildbot/slaves/sidney-fedora3/trunk-sidney-fedora3 (timeout 
1200 secs)
svn: PROPFIND request failed on '/repos/asf/spamassassin/trunk'
svn: PROPFIND of '/repos/asf/spamassassin/trunk': Could not read status 
line: connection was closed by server. (http://svn.apache.org)
program finished with exit code 1

Re: buildbot failure in [...]

Posted by "Malte S. Stretz" <ms...@gmx.net>.
On Friday 17 December 2004 19:14 CET Justin Mason wrote:
> Malte S. Stretz writes:
> > On Friday 17 December 2004 11:36 CET buildbot@bugzilla.spamassassin.org
> >
> > wrote:
> > > The Buildbot has detected a new failure of trunk-debian-stable.
> > >
> > > Buildbot URL: http://bugzilla.spamassassin.org:8010/
> > >
> > > Build Reason: changes
> > > Build Source Stamp: 112
> > > Blamelist: quinlan
> > >
> > > BUILD FAILED: failed svn
> >
> > Those messages are getting a bit annoying, is there any way to filter
> > any builtbot message which contains "BUILD FAILED: failed svn" on the
> > server?
>
> no!
>
> (a) they really are failures.  in this case the svn server seems to have
> died, which is good to know ;)   The whole point of this is to get
> notification of failures.

Hm, if the svn update failed, it's most probably an infra problem or 
something with the buildbot's uplink, something we can't fix anyway and 
which will probably go away by itself.  At least nothing we need a whole 
load of notifies of (the maintainer of the buldbot in question should of 
course get a notice).  IMO :)  But I guess I'll just filter those messages 
locally.

> (b) however the -parker- and -sidney- ones *are* getting annoying. ;)  I
> suggest we turn off those slaves until we can figure out how to get
> buildbot to work with dynamic-IP slaves...

Does anybody know what exactly goes wrong?  Maybe it could work if we use 
port forwarding or stunnel or something to route the traffic to the dynamic 
clients over some server with a static IP?

Cheers,
Malte

-- 
[SGT] Simon G. Tatham: "How to Report Bugs Effectively"
      <http://www.chiark.greenend.org.uk/~sgtatham/bugs.html>
[ESR] Eric S. Raymond: "How To Ask Questions The Smart Way"
      <http://www.catb.org/~esr/faqs/smart-questions.html>

Re: buildbot failure in [...]

Posted by Sidney Markowitz <si...@sidney.com>.
Justin Mason wrote:
> (b) however the -parker- and -sidney- ones *are* getting annoying. ;)  I
> suggest we turn off those slaves until we can figure out how to get
> buildbot to work with dynamic-IP slaves...

I'm running three slaves on one machine, two of them on the same VMWare 
virtual machine and one running native. Most of the time they do not 
generate errors. I have a static ip. The problem cannot be that buildbot 
doesn't work with such a configuration, or else it would never work.

I wonder if svn has trouble with all the clients trying to run at the 
same time on the same physical machine.

  -- sidney

Re: buildbot failure in [...]

Posted by Sidney Markowitz <si...@sidney.com>.
I had a power glitch here which rebooted the server. I think it happened 
in the middle of the svn update causing all three slave jobs to fail, 
and I think that it was a power glitch that caused the reboot. I'm not 
going to bother to bring the buildbot slaves online again before I leave 
on Holiday. The keepalive may have kept them going ok before the power 
failure, but it was too short a timespan to be sure.

  -- sidney

Re: buildbot failure in [...]

Posted by Theo Van Dinter <fe...@kluge.net>.
On Sat, Dec 18, 2004 at 04:31:54PM +0000, Nix wrote:
> don't multiply it by more than, say, ten. If SA goes fifty days without
> a commit something is very wrong. :)

I know nothing about buildbot (haven't had time to look into it), but is there
some way the server can keep track of the commits, and the child can just come
in and query "any work for me?" aka "any work since commit #?"  They can chat
and tests can occur, then when the child is done, it disconnects, waits X,
then connects back and tries again?

It would seem like that would be a bit more efficient than having all clients
constantly connected to the server.

-- 
Randomly Generated Tagline:
echo "Your stdio isn't very std."
              -- Larry Wall in Configure from the perl distribution

Re: buildbot failure in [...]

Posted by Nix <ni...@esperi.org.uk>.
On Fri, 17 Dec 2004, Michael Parker stated:
> I don't think it's so much a dynamic-IP thing as a NAT thing.  Without
> lots of activity (we can go a day or so at a time without a commit)
> the slaves just stop talking to the master and with the connection
> gone behind the NAT the master can't find them.

With most NAT systems worth their salt, this timeout is configurable.

For example, on Linux systems using netfilter (2.4 and 2.6 kernels),
writing a time in seconds to the file
/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_established will
fix it for IPv4 TCP streams: the default value is 432000 (five days).
If you knock it up *too* far, you might see scalability problems, so
don't multiply it by more than, say, ten. If SA goes fifty days without
a commit something is very wrong. :)

-- 
`The sword we forged has turned upon us
 Only now, at the end of all things do we see
 The lamp-bearer dies; only the lamp burns on.'

Re: buildbot failure in [...]

Posted by Michael Parker <pa...@pobox.com>.
On Fri, Dec 17, 2004 at 10:14:16AM -0800, Justin Mason wrote:
> 
> (b) however the -parker- and -sidney- ones *are* getting annoying. ;)  I
> suggest we turn off those slaves until we can figure out how to get
> buildbot to work with dynamic-IP slaves...
> 

I don't think it's so much a dynamic-IP thing as a NAT thing.  Without
lots of activity (we can go a day or so at a time without a commit)
the slaves just stop talking to the master and with the connection
gone behind the NAT the master can't find them.

My tuit supply is limited at the moment, but when I get a chance I was
gonna try to hack up a sort of ping command that could be issued via
cron to keep the connection alive.

Michael