The Windows buildbot saga

[Damjan Jovanovic <da...@apache.org>]

Hi

In build 380, with bootstrapping succeeding and VCVARS32.BAT deleted,
aoo-win7 got far with the build, but apparently hung on "sc deliver" until
it was killed after 20000 seconds with no output.

A subsequent build failed to delete apr/Makefile.win which Infra found
DEVENV had locked again. It's possible that one build.pl instance hung on
that file, while the other compiled everything else, something that only
became apparent when sc delivered and there was nothing else left to
compile. Thus it's not clear whether this is progress.

Another build is running now, and I've asked Infra for the output of "wmic
process list" if it hangs again, which will show command line arguments
passed to dmake/nmake/sh/perl/cl to help us work out what is going on.

Damjan

Re: The Windows buildbot saga

[Damjan Jovanovic <da...@apache.org>]

On Mon, Jul 25, 2016 at 4:26 PM, Damjan Jovanovic <da...@apache.org> wrote:

> Hi
>
> In build 380, with bootstrapping succeeding and VCVARS32.BAT deleted,
> aoo-win7 got far with the build, but apparently hung on "sc deliver" until
> it was killed after 20000 seconds with no output.
>
> A subsequent build failed to delete apr/Makefile.win which Infra found
> DEVENV had locked again. It's possible that one build.pl instance hung on
> that file, while the other compiled everything else, something that only
> became apparent when sc delivered and there was nothing else left to
> compile. Thus it's not clear whether this is progress.
>
>
That is still my best theory.

There was a time when APR did build on the buildbots, and it's not clear
whether it was updates to APR that broke it, or updates to or corruption of
Visual Studio and its tools. DEVENV is strange anyway - on my own setup it
opens a "Visual Studio" window when run from the command line, which does
auto-close and return control to the command line, but only after quite a
while - so eliminating it is a step forward.

APR's Makefile.win builds with either devenv, nmake, or msdev. Without any
options, it tests whether devenv can be found by calling "devenv /help" (an
unsupported command line option btw), and if so, uses it, otherwise falls
back to nmake. In r1754057 I started using USEMAK=1 to force the use of
nmake instead of devenv, but only in r1754110 and r1754165 did I eliminate
the "devenv /help" which was still getting called and apparently still
hanging the build.

With that, and changes to the build script to give bootstrap more time to
do downloads, I did finally get the aoo-win7 buildbot to build in build 401.

Another build is running now, and I've asked Infra for the output of "wmic
> process list" if it hangs again, which will show command line arguments
> passed to dmake/nmake/sh/perl/cl to help us work out what is going on.
>
>
This didn't help, as apparently you have to be administrator for "wmic
process list" to show command line arguments. Infra on HipChat provided a
little help, but generally took long to respond and didn't even answer my
request for remote access. The way I had to hack these buildbots through
their build script and do most of my debugging using educated guesses, is
not something I wish to go through again soon. Is Hudson/Jenkins any better
than buildbot?