You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Oliver-Rainer Wittmann <or...@googlemail.com> on 2013/05/14 10:16:31 UTC
[buildbot] investigate nightly windows build
Hi,
our buildbot building trunk nightly for windows has problems in modul
apr since a couple of days.
Unfortunately, the log does provide nothing for a reason not known to me.
Thus, I will try to investigate the problem.
Hopefully, I can change the buildbot script to get the build output
directly on stdout instead of as html. The html output is currently not
containing the corresponding information about the build of module apr.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 14.05.2013 16:58, Ariel Constenla-Haile wrote:
> Hi Oliver,
>
> On Tue, May 14, 2013 at 10:16:31AM +0200, Oliver-Rainer Wittmann wrote:
>> Hi,
>>
>> our buildbot building trunk nightly for windows has problems in
>> modul apr since a couple of days.
>> Unfortunately, the log does provide nothing for a reason not known to me.
>>
>> Thus, I will try to investigate the problem.
>> Hopefully, I can change the buildbot script to get the build output
>> directly on stdout instead of as html. The html output is currently
>> not containing the corresponding information about the build of
>> module apr.
>
> This seems to be a general problem with the code that copies the logs,
> with the directories outside main, just look at
> http://ci.apache.org/projects/openoffice/buildlogs/linux64/log/unxlngx6.pro.build.html
>
> apr
> apr-util
> cionmp
> hunspell
> serf
>
> all report "No Such Resource File not found."
>
While making the temporary adjustments for further investigations
Herbert and myself noticed that the logs from modules in directory
ext_libraries are not copied.
Thanks Herbert for fixing this.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Ariel Constenla-Haile <ar...@apache.org>.
Hi Oliver,
On Tue, May 14, 2013 at 10:16:31AM +0200, Oliver-Rainer Wittmann wrote:
> Hi,
>
> our buildbot building trunk nightly for windows has problems in
> modul apr since a couple of days.
> Unfortunately, the log does provide nothing for a reason not known to me.
>
> Thus, I will try to investigate the problem.
> Hopefully, I can change the buildbot script to get the build output
> directly on stdout instead of as html. The html output is currently
> not containing the corresponding information about the build of
> module apr.
This seems to be a general problem with the code that copies the logs,
with the directories outside main, just look at
http://ci.apache.org/projects/openoffice/buildlogs/linux64/log/unxlngx6.pro.build.html
apr
apr-util
cionmp
hunspell
serf
all report "No Such Resource File not found."
Regards
--
Ariel Constenla-Haile
La Plata, Argentina
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 17.05.2013 01:57, Andrew Rist wrote:
>
>>>>>
>>>>> our buildbot building trunk nightly for windows has problems in modul
>>>>> apr since a couple of days.
>>>>> Unfortunately, the log does provide nothing for a reason not known to
>>>>> me.
>>>>>
>>>>> Thus, I will try to investigate the problem.
>>>>> Hopefully, I can change the buildbot script to get the build output
>>>>> directly on stdout instead of as html. The html output is currently
>>>>> not
>>>>> containing the corresponding information about the build of module
>>>>> apr.
>>>>>
>>>>
>>>> Done and Thx to Herbert triggering a clean build.
>>>> Unfortunately, the build was successful. Thus, I assume the reason
>>>> that we had no nightly windows builds from trunk since 2013-04-28 was
>>>> that no clean build had been performed.
>>> This is not the case. The clean build is not the panacea you see it
>>> as. As mentioned in several other communications, I went onto the box
>>> and cleaned up some processes that were hung (win7, win7snap, and
>>> win7ia2). All built successfully - even though the other two were
>>> incremental. The hung processes tend to occur /more/ during clean
>>> builds - not all the time, just more often. Thus, clean builds are more
>>> likely to create this type of build failure, they are not a fix as
>>> you're suggesting.
>>>
>>
>> Thanks for the information.
>> In order to have something more tangible for fixing this defect of
>> hanging build processes I propose to start an corresponding
>> investigation.
> Sounds good...
thx, for the +1
>> At least we should have a look after each build, esp. after each clean
>> build, if there are processes which hang.
>> Andrew, can only you perform such an investigation, because (as far as
>> I know) you are the only who have direct access on the machine?
> This I am not signing up for - I can't really commit to having the time
> to focus on this. I will help interacting with infra to get access for
> any committer that wants to take this on..
> (I will look next time we get one of these and at least specifically
> identify what process is hanging and post that back, but I am afraid the
> debugging of the stack is going to be a bit more involved)
Sorry, again a misunderstanding.
I just wanted to know, if currently only you had the karma to perform
such an investigation - I did not meant that you have to perform it.
Thus, it would be great (and needed), that one or two or three
volunteers show up in order to take responsibility here.
Thanks to Andrew for the offer to interact with infra for getting the
karma for these volunteers.
Who has interest to take responsibility here?
Best regards, Oliver.
> A.
>>
>> Best regards, Oliver.
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Andrew Rist <an...@oracle.com>.
On 5/16/2013 1:24 AM, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 15.05.2013 21:51, Andrew Rist wrote:
>>
>> On 5/15/2013 12:03 AM, Oliver-Rainer Wittmann wrote:
>>> Hi,
>>>
>>> On 14.05.2013 10:16, Oliver-Rainer Wittmann wrote:
>>>> Hi,
>>>>
>>>> our buildbot building trunk nightly for windows has problems in modul
>>>> apr since a couple of days.
>>>> Unfortunately, the log does provide nothing for a reason not known to
>>>> me.
>>>>
>>>> Thus, I will try to investigate the problem.
>>>> Hopefully, I can change the buildbot script to get the build output
>>>> directly on stdout instead of as html. The html output is currently
>>>> not
>>>> containing the corresponding information about the build of module
>>>> apr.
>>>>
>>>
>>> Done and Thx to Herbert triggering a clean build.
>>> Unfortunately, the build was successful. Thus, I assume the reason
>>> that we had no nightly windows builds from trunk since 2013-04-28 was
>>> that no clean build had been performed.
>> This is not the case. The clean build is not the panacea you see it
>> as. As mentioned in several other communications, I went onto the box
>> and cleaned up some processes that were hung (win7, win7snap, and
>> win7ia2). All built successfully - even though the other two were
>> incremental. The hung processes tend to occur /more/ during clean
>> builds - not all the time, just more often. Thus, clean builds are more
>> likely to create this type of build failure, they are not a fix as
>> you're suggesting.
>>
>
> Thanks for the information.
> In order to have something more tangible for fixing this defect of
> hanging build processes I propose to start an corresponding
> investigation.
Sounds good...
> At least we should have a look after each build, esp. after each clean
> build, if there are processes which hang.
> Andrew, can only you perform such an investigation, because (as far as
> I know) you are the only who have direct access on the machine?
This I am not signing up for - I can't really commit to having the time
to focus on this. I will help interacting with infra to get access for
any committer that wants to take this on..
(I will look next time we get one of these and at least specifically
identify what process is hanging and post that back, but I am afraid the
debugging of the stack is going to be a bit more involved)
A.
>
> Best regards, Oliver.
>
>
>>
>> A.
>>
>> (and good catch on the ext_source logs...)
>>>
>>> I am now reverting my temporary changes for the buildbot aoo-win7.
>>>
>>> Best regards, Oliver.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 31.05.2013 01:28, Andrew Rist wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> [snip]
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have seen that #621 of aoo-win7 also had this error and
>>>>>>>>>>> that you
>>>>>>>>>>> had
>>>>>>>>>>> again cleaned up the hanging processes.
>>>>>>>>>>>
>>>>>>>>>>> In order to get some error output I will switch off the HTML
>>>>>>>>>>> output
>>>>>>>>>>> (no
>>>>>>>>>>> "--html" option) and the multiprocessor build (no "-P2 -- -P2"
>>>>>>>>>>> options)
>>>>>>>>>>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2
>>>>>>>>>>> build
>>>>>>>>>>> which is always clean.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Build #113 of aoo-w7ia2 went well.
>>>>>>>>>> The failure occured just because I forgot to disable the
>>>>>>>>>> collection of
>>>>>>>>>> the HTML build logs.
>>>>>>>>> I'm confused - was there something wrong with the w7ia2 build. The
>>>>>>>>> previous two builds were clean:
>>>>>>>>> http://ci.apache.org/builders/aoo-w7ia2
>>>>>>>>> Other than stumbling over hung processes, I don't think there
>>>>>>>>> is an
>>>>>>>>> issue with ia2.
>>>>>>>>
>>>>>>>> We had observed the problem of the hanging process.
>>>>>>>> #105 had hanging process
>>>>>>>> #106 went well after your process cleaning
>>>>>>>> #107, #108, #109, #110 had again problems with hanging processes
>>>>>>>> #111 went well after your process cleaning
>>>>>>>> #112 the first successful build without your invention
>>>>>>>> #113 went well with my first temporary investigation stuff -
>>>>>>>> failure
>>>>>>>> just because of error in copying non-existing HTML logs.
>>>>>>>> #114 went well with my second investigation stuff
>>>>>>>>
>>>>>>>> I decided to do some investigation stuff as the problem showed
>>>>>>>> up two
>>>>>>>> times in the last days. Currently, we do not hit the problem. But
>>>>>>>> when
>>>>>>>> it occurs again, we will have some more information due to the
>>>>>>>> different
>>>>>>>> logging. The problem of the current HTML logging is that we do not
>>>>>>>> have
>>>>>>>> it, when the hanging process error occurs.
>>>>>>>>
>>>>>>>> As #114 went well I will now switch on the default multiprocesses
>>>>>>>> builds
>>>>>>>> - 2 modules built in parallel with each using 2 make/dmake
>>>>>>>> processes.
>>>>>>>>
>>>>>>>
>>>>>>> #115 build of aoo-w7ia2 having standard multiprocessor builds
>>>>>>> (options
>>>>>>> "-P2 -- -P2") went well.
>>>>>>> I will leave the HTML output switched off until Monday. May be the
>>>>>>> problem with the hanging process occurs. Then we have some output
>>>>>>> when
>>>>>>> and where it occurs.
>>>>>>
>>>>>> #117 build of aoo-w7ia2 showed again the error.
>>>>>> Thus, following #118 build fails - it could not remove
>>>>>> /ext_libraries/apr/wntmcsi12/misc/apr-1.4.5/Makefile.win
>>>>>>
>>>>>
>>>>> Andrew, can you please kill the hanging processes.
>>>> Cleaned up - fingers crossed that this works. great work.
>>>
>>> Thanks for the cleaning - #120 build of aoo-w7ia2 went well
>>>
>>> I will switch on the HTML output for aoo-w7ia2 again.
>>> I will mark the log collection task to be always performed - may be this
>>> will give us HTML logs even when a failure occurs.
>>>
>>
>> The Bad: #121 build of aoo-w7ia2 failed again.
>> The Good: The HTML log collection in case of such a failure works.
>>
>> The HTML logs of #121 (which will be overwritten by the next build)
>> reveals that modules apr and sc were built successfully, but the
>> 'deliver' steps were not performed. I assume this is again caused by
>> one (or more) hanging process(es)
>>
>> @Andrew:
>> When you have time, please have a look at the machine. If possible,
>> please provide information about the hanging processes in case they
>> are existing and causing again the failure.
> hung process:
> cl /nologo /?
> with a parent process of:
> C:\cygwin\bin\sh.exe -c "dmake -P2 verbose=true > [long path]/apr.txt 2>&1"
> with the parent process of:
> [long path]\build.pl --all --html -P -- -P
>
> (all I need to clean up is to kill the cl process, and the others
> eventually exit)
> (and it's cleaned up now)
First, thanks again for the clean up.
Second, why is there still a "cl /nologo /?" process?
Because, I am too stupid. I have made the change to the apr makefile
(removing the call to this process) on trunk and did not integrate this
change into branch ia2.
I will catch up on it next week.
> A.
>
>
>
>>
>> As nobody else showed up I am volunteering to take care for our
>> Windows buildbots. Andrew, can you support me to get the corresponding
>> karma to have direct access to the machine(s) hosting our Windows
>> buildbots?
> I'll look for you coming online, and we can do that over IRC
Did you mean the IRC from ASF infra?
If we want to do it via IRC (via the ASF Infra one or another one) I
think we need to make an appointment. In general our working hours are
more or less not overlapping.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Andrew Rist <an...@oracle.com>.
On 5/30/2013 6:04 AM, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 29.05.2013 10:35, Oliver-Rainer Wittmann wrote:
>>>>>>>>>>>
>>>>>>>>>>> [snip]
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have seen that #621 of aoo-win7 also had this error and
>>>>>>>>>> that you
>>>>>>>>>> had
>>>>>>>>>> again cleaned up the hanging processes.
>>>>>>>>>>
>>>>>>>>>> In order to get some error output I will switch off the HTML
>>>>>>>>>> output
>>>>>>>>>> (no
>>>>>>>>>> "--html" option) and the multiprocessor build (no "-P2 -- -P2"
>>>>>>>>>> options)
>>>>>>>>>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2
>>>>>>>>>> build
>>>>>>>>>> which is always clean.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Build #113 of aoo-w7ia2 went well.
>>>>>>>>> The failure occured just because I forgot to disable the
>>>>>>>>> collection of
>>>>>>>>> the HTML build logs.
>>>>>>>> I'm confused - was there something wrong with the w7ia2 build.
>>>>>>>> The
>>>>>>>> previous two builds were clean:
>>>>>>>> http://ci.apache.org/builders/aoo-w7ia2
>>>>>>>> Other than stumbling over hung processes, I don't think there
>>>>>>>> is an
>>>>>>>> issue with ia2.
>>>>>>>
>>>>>>> We had observed the problem of the hanging process.
>>>>>>> #105 had hanging process
>>>>>>> #106 went well after your process cleaning
>>>>>>> #107, #108, #109, #110 had again problems with hanging processes
>>>>>>> #111 went well after your process cleaning
>>>>>>> #112 the first successful build without your invention
>>>>>>> #113 went well with my first temporary investigation stuff -
>>>>>>> failure
>>>>>>> just because of error in copying non-existing HTML logs.
>>>>>>> #114 went well with my second investigation stuff
>>>>>>>
>>>>>>> I decided to do some investigation stuff as the problem showed
>>>>>>> up two
>>>>>>> times in the last days. Currently, we do not hit the problem. But
>>>>>>> when
>>>>>>> it occurs again, we will have some more information due to the
>>>>>>> different
>>>>>>> logging. The problem of the current HTML logging is that we do not
>>>>>>> have
>>>>>>> it, when the hanging process error occurs.
>>>>>>>
>>>>>>> As #114 went well I will now switch on the default multiprocesses
>>>>>>> builds
>>>>>>> - 2 modules built in parallel with each using 2 make/dmake
>>>>>>> processes.
>>>>>>>
>>>>>>
>>>>>> #115 build of aoo-w7ia2 having standard multiprocessor builds
>>>>>> (options
>>>>>> "-P2 -- -P2") went well.
>>>>>> I will leave the HTML output switched off until Monday. May be the
>>>>>> problem with the hanging process occurs. Then we have some output
>>>>>> when
>>>>>> and where it occurs.
>>>>>
>>>>> #117 build of aoo-w7ia2 showed again the error.
>>>>> Thus, following #118 build fails - it could not remove
>>>>> /ext_libraries/apr/wntmcsi12/misc/apr-1.4.5/Makefile.win
>>>>>
>>>>
>>>> Andrew, can you please kill the hanging processes.
>>> Cleaned up - fingers crossed that this works. great work.
>>
>> Thanks for the cleaning - #120 build of aoo-w7ia2 went well
>>
>> I will switch on the HTML output for aoo-w7ia2 again.
>> I will mark the log collection task to be always performed - may be this
>> will give us HTML logs even when a failure occurs.
>>
>
> The Bad: #121 build of aoo-w7ia2 failed again.
> The Good: The HTML log collection in case of such a failure works.
>
> The HTML logs of #121 (which will be overwritten by the next build)
> reveals that modules apr and sc were built successfully, but the
> 'deliver' steps were not performed. I assume this is again caused by
> one (or more) hanging process(es)
>
> @Andrew:
> When you have time, please have a look at the machine. If possible,
> please provide information about the hanging processes in case they
> are existing and causing again the failure.
hung process:
cl /nologo /?
with a parent process of:
C:\cygwin\bin\sh.exe -c "dmake -P2 verbose=true > [long path]/apr.txt 2>&1"
with the parent process of:
[long path]\build.pl --all --html -P -- -P
(all I need to clean up is to kill the cl process, and the others
eventually exit)
(and it's cleaned up now)
A.
>
> As nobody else showed up I am volunteering to take care for our
> Windows buildbots. Andrew, can you support me to get the corresponding
> karma to have direct access to the machine(s) hosting our Windows
> buildbots?
I'll look for you coming online, and we can do that over IRC
>
> Thanks in advance.
>
>
> Best regards, Oliver.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 29.05.2013 10:35, Oliver-Rainer Wittmann wrote:
>>>>>>>>>>
>>>>>>>>>> [snip]
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have seen that #621 of aoo-win7 also had this error and that you
>>>>>>>>> had
>>>>>>>>> again cleaned up the hanging processes.
>>>>>>>>>
>>>>>>>>> In order to get some error output I will switch off the HTML
>>>>>>>>> output
>>>>>>>>> (no
>>>>>>>>> "--html" option) and the multiprocessor build (no "-P2 -- -P2"
>>>>>>>>> options)
>>>>>>>>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2
>>>>>>>>> build
>>>>>>>>> which is always clean.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Build #113 of aoo-w7ia2 went well.
>>>>>>>> The failure occured just because I forgot to disable the
>>>>>>>> collection of
>>>>>>>> the HTML build logs.
>>>>>>> I'm confused - was there something wrong with the w7ia2 build. The
>>>>>>> previous two builds were clean:
>>>>>>> http://ci.apache.org/builders/aoo-w7ia2
>>>>>>> Other than stumbling over hung processes, I don't think there is an
>>>>>>> issue with ia2.
>>>>>>
>>>>>> We had observed the problem of the hanging process.
>>>>>> #105 had hanging process
>>>>>> #106 went well after your process cleaning
>>>>>> #107, #108, #109, #110 had again problems with hanging processes
>>>>>> #111 went well after your process cleaning
>>>>>> #112 the first successful build without your invention
>>>>>> #113 went well with my first temporary investigation stuff - failure
>>>>>> just because of error in copying non-existing HTML logs.
>>>>>> #114 went well with my second investigation stuff
>>>>>>
>>>>>> I decided to do some investigation stuff as the problem showed up two
>>>>>> times in the last days. Currently, we do not hit the problem. But
>>>>>> when
>>>>>> it occurs again, we will have some more information due to the
>>>>>> different
>>>>>> logging. The problem of the current HTML logging is that we do not
>>>>>> have
>>>>>> it, when the hanging process error occurs.
>>>>>>
>>>>>> As #114 went well I will now switch on the default multiprocesses
>>>>>> builds
>>>>>> - 2 modules built in parallel with each using 2 make/dmake processes.
>>>>>>
>>>>>
>>>>> #115 build of aoo-w7ia2 having standard multiprocessor builds (options
>>>>> "-P2 -- -P2") went well.
>>>>> I will leave the HTML output switched off until Monday. May be the
>>>>> problem with the hanging process occurs. Then we have some output when
>>>>> and where it occurs.
>>>>
>>>> #117 build of aoo-w7ia2 showed again the error.
>>>> Thus, following #118 build fails - it could not remove
>>>> /ext_libraries/apr/wntmcsi12/misc/apr-1.4.5/Makefile.win
>>>>
>>>
>>> Andrew, can you please kill the hanging processes.
>> Cleaned up - fingers crossed that this works. great work.
>
> Thanks for the cleaning - #120 build of aoo-w7ia2 went well
>
> I will switch on the HTML output for aoo-w7ia2 again.
> I will mark the log collection task to be always performed - may be this
> will give us HTML logs even when a failure occurs.
>
The Bad: #121 build of aoo-w7ia2 failed again.
The Good: The HTML log collection in case of such a failure works.
The HTML logs of #121 (which will be overwritten by the next build)
reveals that modules apr and sc were built successfully, but the
'deliver' steps were not performed. I assume this is again caused by one
(or more) hanging process(es)
@Andrew:
When you have time, please have a look at the machine. If possible,
please provide information about the hanging processes in case they are
existing and causing again the failure.
As nobody else showed up I am volunteering to take care for our Windows
buildbots. Andrew, can you support me to get the corresponding karma to
have direct access to the machine(s) hosting our Windows buildbots?
Thanks in advance.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 28.05.2013 19:17, Andrew Rist wrote:
>
> On 5/27/2013 3:02 AM, Oliver-Rainer Wittmann wrote:
>> Hi,
>>
>> On 27.05.2013 12:00, Oliver-Rainer Wittmann wrote:
>>>>>>>>>
>>>>>>>>> [snip]
>>>>>>>>>
>>>>>>>>
>>>>>>>> I have seen that #621 of aoo-win7 also had this error and that you
>>>>>>>> had
>>>>>>>> again cleaned up the hanging processes.
>>>>>>>>
>>>>>>>> In order to get some error output I will switch off the HTML output
>>>>>>>> (no
>>>>>>>> "--html" option) and the multiprocessor build (no "-P2 -- -P2"
>>>>>>>> options)
>>>>>>>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2
>>>>>>>> build
>>>>>>>> which is always clean.
>>>>>>>>
>>>>>>>
>>>>>>> Build #113 of aoo-w7ia2 went well.
>>>>>>> The failure occured just because I forgot to disable the
>>>>>>> collection of
>>>>>>> the HTML build logs.
>>>>>> I'm confused - was there something wrong with the w7ia2 build. The
>>>>>> previous two builds were clean:
>>>>>> http://ci.apache.org/builders/aoo-w7ia2
>>>>>> Other than stumbling over hung processes, I don't think there is an
>>>>>> issue with ia2.
>>>>>
>>>>> We had observed the problem of the hanging process.
>>>>> #105 had hanging process
>>>>> #106 went well after your process cleaning
>>>>> #107, #108, #109, #110 had again problems with hanging processes
>>>>> #111 went well after your process cleaning
>>>>> #112 the first successful build without your invention
>>>>> #113 went well with my first temporary investigation stuff - failure
>>>>> just because of error in copying non-existing HTML logs.
>>>>> #114 went well with my second investigation stuff
>>>>>
>>>>> I decided to do some investigation stuff as the problem showed up two
>>>>> times in the last days. Currently, we do not hit the problem. But when
>>>>> it occurs again, we will have some more information due to the
>>>>> different
>>>>> logging. The problem of the current HTML logging is that we do not
>>>>> have
>>>>> it, when the hanging process error occurs.
>>>>>
>>>>> As #114 went well I will now switch on the default multiprocesses
>>>>> builds
>>>>> - 2 modules built in parallel with each using 2 make/dmake processes.
>>>>>
>>>>
>>>> #115 build of aoo-w7ia2 having standard multiprocessor builds (options
>>>> "-P2 -- -P2") went well.
>>>> I will leave the HTML output switched off until Monday. May be the
>>>> problem with the hanging process occurs. Then we have some output when
>>>> and where it occurs.
>>>
>>> #117 build of aoo-w7ia2 showed again the error.
>>> Thus, following #118 build fails - it could not remove
>>> /ext_libraries/apr/wntmcsi12/misc/apr-1.4.5/Makefile.win
>>>
>>
>> Andrew, can you please kill the hanging processes.
> Cleaned up - fingers crossed that this works. great work.
Thanks for the cleaning - #120 build of aoo-w7ia2 went well
I will switch on the HTML output for aoo-w7ia2 again.
I will mark the log collection task to be always performed - may be this
will give us HTML logs even when a failure occurs.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Andrew Rist <an...@oracle.com>.
On 5/27/2013 3:02 AM, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 27.05.2013 12:00, Oliver-Rainer Wittmann wrote:
>>>>>>>>
>>>>>>>> [snip]
>>>>>>>>
>>>>>>>
>>>>>>> I have seen that #621 of aoo-win7 also had this error and that you
>>>>>>> had
>>>>>>> again cleaned up the hanging processes.
>>>>>>>
>>>>>>> In order to get some error output I will switch off the HTML output
>>>>>>> (no
>>>>>>> "--html" option) and the multiprocessor build (no "-P2 -- -P2"
>>>>>>> options)
>>>>>>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2
>>>>>>> build
>>>>>>> which is always clean.
>>>>>>>
>>>>>>
>>>>>> Build #113 of aoo-w7ia2 went well.
>>>>>> The failure occured just because I forgot to disable the
>>>>>> collection of
>>>>>> the HTML build logs.
>>>>> I'm confused - was there something wrong with the w7ia2 build. The
>>>>> previous two builds were clean:
>>>>> http://ci.apache.org/builders/aoo-w7ia2
>>>>> Other than stumbling over hung processes, I don't think there is an
>>>>> issue with ia2.
>>>>
>>>> We had observed the problem of the hanging process.
>>>> #105 had hanging process
>>>> #106 went well after your process cleaning
>>>> #107, #108, #109, #110 had again problems with hanging processes
>>>> #111 went well after your process cleaning
>>>> #112 the first successful build without your invention
>>>> #113 went well with my first temporary investigation stuff - failure
>>>> just because of error in copying non-existing HTML logs.
>>>> #114 went well with my second investigation stuff
>>>>
>>>> I decided to do some investigation stuff as the problem showed up two
>>>> times in the last days. Currently, we do not hit the problem. But when
>>>> it occurs again, we will have some more information due to the
>>>> different
>>>> logging. The problem of the current HTML logging is that we do not
>>>> have
>>>> it, when the hanging process error occurs.
>>>>
>>>> As #114 went well I will now switch on the default multiprocesses
>>>> builds
>>>> - 2 modules built in parallel with each using 2 make/dmake processes.
>>>>
>>>
>>> #115 build of aoo-w7ia2 having standard multiprocessor builds (options
>>> "-P2 -- -P2") went well.
>>> I will leave the HTML output switched off until Monday. May be the
>>> problem with the hanging process occurs. Then we have some output when
>>> and where it occurs.
>>
>> #117 build of aoo-w7ia2 showed again the error.
>> Thus, following #118 build fails - it could not remove
>> /ext_libraries/apr/wntmcsi12/misc/apr-1.4.5/Makefile.win
>>
>
> Andrew, can you please kill the hanging processes.
Cleaned up - fingers crossed that this works. great work.
A.
> Many thanks in advance.
>
> Best regards, Oliver.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 27.05.2013 12:00, Oliver-Rainer Wittmann wrote:
>>>>>>>
>>>>>>> [snip]
>>>>>>>
>>>>>>
>>>>>> I have seen that #621 of aoo-win7 also had this error and that you
>>>>>> had
>>>>>> again cleaned up the hanging processes.
>>>>>>
>>>>>> In order to get some error output I will switch off the HTML output
>>>>>> (no
>>>>>> "--html" option) and the multiprocessor build (no "-P2 -- -P2"
>>>>>> options)
>>>>>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2 build
>>>>>> which is always clean.
>>>>>>
>>>>>
>>>>> Build #113 of aoo-w7ia2 went well.
>>>>> The failure occured just because I forgot to disable the collection of
>>>>> the HTML build logs.
>>>> I'm confused - was there something wrong with the w7ia2 build. The
>>>> previous two builds were clean: http://ci.apache.org/builders/aoo-w7ia2
>>>> Other than stumbling over hung processes, I don't think there is an
>>>> issue with ia2.
>>>
>>> We had observed the problem of the hanging process.
>>> #105 had hanging process
>>> #106 went well after your process cleaning
>>> #107, #108, #109, #110 had again problems with hanging processes
>>> #111 went well after your process cleaning
>>> #112 the first successful build without your invention
>>> #113 went well with my first temporary investigation stuff - failure
>>> just because of error in copying non-existing HTML logs.
>>> #114 went well with my second investigation stuff
>>>
>>> I decided to do some investigation stuff as the problem showed up two
>>> times in the last days. Currently, we do not hit the problem. But when
>>> it occurs again, we will have some more information due to the different
>>> logging. The problem of the current HTML logging is that we do not have
>>> it, when the hanging process error occurs.
>>>
>>> As #114 went well I will now switch on the default multiprocesses builds
>>> - 2 modules built in parallel with each using 2 make/dmake processes.
>>>
>>
>> #115 build of aoo-w7ia2 having standard multiprocessor builds (options
>> "-P2 -- -P2") went well.
>> I will leave the HTML output switched off until Monday. May be the
>> problem with the hanging process occurs. Then we have some output when
>> and where it occurs.
>
> #117 build of aoo-w7ia2 showed again the error.
> Thus, following #118 build fails - it could not remove
> /ext_libraries/apr/wntmcsi12/misc/apr-1.4.5/Makefile.win
>
Andrew, can you please kill the hanging processes.
Many thanks in advance.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 24.05.2013 10:21, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 23.05.2013 09:19, Oliver-Rainer Wittmann wrote:
>>>>>>
>>>>>> [snip]
>>>>>>
>>>>>
>>>>> I have seen that #621 of aoo-win7 also had this error and that you had
>>>>> again cleaned up the hanging processes.
>>>>>
>>>>> In order to get some error output I will switch off the HTML output
>>>>> (no
>>>>> "--html" option) and the multiprocessor build (no "-P2 -- -P2"
>>>>> options)
>>>>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2 build
>>>>> which is always clean.
>>>>>
>>>>
>>>> Build #113 of aoo-w7ia2 went well.
>>>> The failure occured just because I forgot to disable the collection of
>>>> the HTML build logs.
>>> I'm confused - was there something wrong with the w7ia2 build. The
>>> previous two builds were clean: http://ci.apache.org/builders/aoo-w7ia2
>>> Other than stumbling over hung processes, I don't think there is an
>>> issue with ia2.
>>
>> We had observed the problem of the hanging process.
>> #105 had hanging process
>> #106 went well after your process cleaning
>> #107, #108, #109, #110 had again problems with hanging processes
>> #111 went well after your process cleaning
>> #112 the first successful build without your invention
>> #113 went well with my first temporary investigation stuff - failure
>> just because of error in copying non-existing HTML logs.
>> #114 went well with my second investigation stuff
>>
>> I decided to do some investigation stuff as the problem showed up two
>> times in the last days. Currently, we do not hit the problem. But when
>> it occurs again, we will have some more information due to the different
>> logging. The problem of the current HTML logging is that we do not have
>> it, when the hanging process error occurs.
>>
>> As #114 went well I will now switch on the default multiprocesses builds
>> - 2 modules built in parallel with each using 2 make/dmake processes.
>>
>
> #115 build of aoo-w7ia2 having standard multiprocessor builds (options
> "-P2 -- -P2") went well.
> I will leave the HTML output switched off until Monday. May be the
> problem with the hanging process occurs. Then we have some output when
> and where it occurs.
#117 build of aoo-w7ia2 showed again the error.
Thus, following #118 build fails - it could not remove
/ext_libraries/apr/wntmcsi12/misc/apr-1.4.5/Makefile.win
A deeper look into the #117 build output reveals the following:
- module apr was successful build, but the 'deliver' was not performed.
While searching for the reason Andre and myself again consulted Andrew's
information about the hanging processes. Andrew mentioned process
"cl.exe - cl /nologo /?". We search for the corresponding code which
starts such a process. We found it inside the makefile uses for the
Windows apr build - namely
/ext_libraries/apr/wntmcsi12/misc/apr-1.4.5/Makefile.win. The process
seems to be used to determine, if a 64bit compiler should be used or not.
As we have a 32bit compiler for our Windows platform, I will simplify
this makefile.
Hopefully this will solve our buildbot problems.
>
> For the coming clean build of aoo-win7 at the weekend I will switch off
> its HTML output in order to have some output in case the problem with
> the hanging process occurs.
clean build of aoo-win7 went well. Thus, I reverted the temporary switch
off of the HTML output.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 23.05.2013 09:19, Oliver-Rainer Wittmann wrote:
>>>>>
>>>>> [snip]
>>>>>
>>>>
>>>> I have seen that #621 of aoo-win7 also had this error and that you had
>>>> again cleaned up the hanging processes.
>>>>
>>>> In order to get some error output I will switch off the HTML output (no
>>>> "--html" option) and the multiprocessor build (no "-P2 -- -P2" options)
>>>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2 build
>>>> which is always clean.
>>>>
>>>
>>> Build #113 of aoo-w7ia2 went well.
>>> The failure occured just because I forgot to disable the collection of
>>> the HTML build logs.
>> I'm confused - was there something wrong with the w7ia2 build. The
>> previous two builds were clean: http://ci.apache.org/builders/aoo-w7ia2
>> Other than stumbling over hung processes, I don't think there is an
>> issue with ia2.
>
> We had observed the problem of the hanging process.
> #105 had hanging process
> #106 went well after your process cleaning
> #107, #108, #109, #110 had again problems with hanging processes
> #111 went well after your process cleaning
> #112 the first successful build without your invention
> #113 went well with my first temporary investigation stuff - failure
> just because of error in copying non-existing HTML logs.
> #114 went well with my second investigation stuff
>
> I decided to do some investigation stuff as the problem showed up two
> times in the last days. Currently, we do not hit the problem. But when
> it occurs again, we will have some more information due to the different
> logging. The problem of the current HTML logging is that we do not have
> it, when the hanging process error occurs.
>
> As #114 went well I will now switch on the default multiprocesses builds
> - 2 modules built in parallel with each using 2 make/dmake processes.
>
#115 build of aoo-w7ia2 having standard multiprocessor builds (options
"-P2 -- -P2") went well.
I will leave the HTML output switched off until Monday. May be the
problem with the hanging process occurs. Then we have some output when
and where it occurs.
For the coming clean build of aoo-win7 at the weekend I will switch off
its HTML output in order to have some output in case the problem with
the hanging process occurs.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 22.05.2013 20:11, Andrew Rist wrote:
>
> On 5/22/2013 3:30 AM, Oliver-Rainer Wittmann wrote:
>> Hi,
>>
>> On 21.05.2013 09:53, Oliver-Rainer Wittmann wrote:
>>> Hi,
>>>
>>> On 17.05.2013 23:11, Andrew Rist wrote:
>>>>
>>>> [snip]
>>>>
>>>
>>> I have seen that #621 of aoo-win7 also had this error and that you had
>>> again cleaned up the hanging processes.
>>>
>>> In order to get some error output I will switch off the HTML output (no
>>> "--html" option) and the multiprocessor build (no "-P2 -- -P2" options)
>>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2 build
>>> which is always clean.
>>>
>>
>> Build #113 of aoo-w7ia2 went well.
>> The failure occured just because I forgot to disable the collection of
>> the HTML build logs.
> I'm confused - was there something wrong with the w7ia2 build. The
> previous two builds were clean: http://ci.apache.org/builders/aoo-w7ia2
> Other than stumbling over hung processes, I don't think there is an
> issue with ia2.
We had observed the problem of the hanging process.
#105 had hanging process
#106 went well after your process cleaning
#107, #108, #109, #110 had again problems with hanging processes
#111 went well after your process cleaning
#112 the first successful build without your invention
#113 went well with my first temporary investigation stuff - failure
just because of error in copying non-existing HTML logs.
#114 went well with my second investigation stuff
I decided to do some investigation stuff as the problem showed up two
times in the last days. Currently, we do not hit the problem. But when
it occurs again, we will have some more information due to the different
logging. The problem of the current HTML logging is that we do not have
it, when the hanging process error occurs.
As #114 went well I will now switch on the default multiprocesses builds
- 2 modules built in parallel with each using 2 make/dmake processes.
Best regards, Oliver.
> A.
>
>
>>
>> May be the former errors are related to the multiprocessor build.
>> I will switch on the multiprocessor build inside each module - "--
>> -P2" option for the next build.
>>
>> Best regards, Oliver.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Andrew Rist <an...@oracle.com>.
On 5/22/2013 3:30 AM, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 21.05.2013 09:53, Oliver-Rainer Wittmann wrote:
>> Hi,
>>
>> On 17.05.2013 23:11, Andrew Rist wrote:
>>>
>>> [snip]
>>>
>>
>> I have seen that #621 of aoo-win7 also had this error and that you had
>> again cleaned up the hanging processes.
>>
>> In order to get some error output I will switch off the HTML output (no
>> "--html" option) and the multiprocessor build (no "-P2 -- -P2" options)
>> before the weekly clean build of aoo-win7 and for the aoo-w7ia2 build
>> which is always clean.
>>
>
> Build #113 of aoo-w7ia2 went well.
> The failure occured just because I forgot to disable the collection of
> the HTML build logs.
I'm confused - was there something wrong with the w7ia2 build. The
previous two builds were clean: http://ci.apache.org/builders/aoo-w7ia2
Other than stumbling over hung processes, I don't think there is an
issue with ia2.
A.
>
> May be the former errors are related to the multiprocessor build.
> I will switch on the multiprocessor build inside each module - "--
> -P2" option for the next build.
>
> Best regards, Oliver.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
--
Andrew Rist | Interoperability Architect
OracleCorporate Architecture Group
Redwood Shores, CA | 650.506.9847
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 21.05.2013 09:53, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 17.05.2013 23:11, Andrew Rist wrote:
>>
>> [snip]
>>
>
> I have seen that #621 of aoo-win7 also had this error and that you had
> again cleaned up the hanging processes.
>
> In order to get some error output I will switch off the HTML output (no
> "--html" option) and the multiprocessor build (no "-P2 -- -P2" options)
> before the weekly clean build of aoo-win7 and for the aoo-w7ia2 build
> which is always clean.
>
Build #113 of aoo-w7ia2 went well.
The failure occured just because I forgot to disable the collection of
the HTML build logs.
May be the former errors are related to the multiprocessor build.
I will switch on the multiprocessor build inside each module - "-- -P2"
option for the next build.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 17.05.2013 23:11, Andrew Rist wrote:
>
> [snip]
>
>>>>
>>>> It seems that this defect just occured with build #105 of aoo-w7ia2 -
>>>> see [1]. The build had been "killed". I assume that the one or the
>>>> other process of this build is still working.
>>>> Can somebody with corresponding karma check, if there are again
>>>> hanging processes?
>>>>
>>>> [1] http://ci.apache.org/builders/aoo-w7ia2/builds/105
>>> The build was not "killed" - the process that was running didn't report
>>> back in 12000 sec = 200 min or 3+hours
>>> At that point the buildbot tries to clean up, but this is the reaction,
>>> not the root cause.
>>>
>>> command timed out: 12000 seconds without output, killing pid 2472
>>> SIGKILL failed to kill process
>>> using fake rc=-1
>>> program finished with exit code -1
>>>
>>
>> That is what I meant by "killed" - sorry for not expressing myself clear.
>>
>>> When I checked it later, the process was still hung (thus it's unlikely
>>> that our problem is just with the length of the timeout). This is what
>>> one of these hung processes looks like, and any subsequent builds will
>>> fail if it's not cleaned up, as the processes lock files and block
>>> subsequent compiles of the same package.
>>>
>>
>> Thanks for having a look.
>> Build #106 of aoo-w7ia2 went well after your clean up.
>> Unfortunately, build #107 of aoo-w7ia2 had again the same failure as
>> build #105.
> ok - so I am now killing the hung processes - here is what I find:
>
> * cl.exe - cl /nologo /?
> * sh.exe - C:\cygwin\bin\sh.exe -c "dmake -P2 verbose=true >
> /cygdrive/.../apr.txt 2>&1"
> * perl.exe - C:\cygwin\bin\perl.exe E:/.../build.pl --all --html -P2
> -- -P2
>
> when I kill the cl process the other two come to life - i.e. that's the
> hung process
> looking in to the html progress page, the build is now finishing without
> reporting errors.
>
Thanks for the information.
I have seen that #621 of aoo-win7 also had this error and that you had
again cleaned up the hanging processes.
In order to get some error output I will switch off the HTML output (no
"--html" option) and the multiprocessor build (no "-P2 -- -P2" options)
before the weekly clean build of aoo-win7 and for the aoo-win7-ia2 build
which is always clean.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Andrew Rist <an...@oracle.com>.
On 5/17/2013 1:53 AM, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 17.05.2013 01:53, Andrew Rist wrote:
>>
>>>>>>>
>>>>>>> our buildbot building trunk nightly for windows has problems in
>>>>>>> modul
>>>>>>> apr since a couple of days.
>>>>>>> Unfortunately, the log does provide nothing for a reason not
>>>>>>> known to
>>>>>>> me.
>>>>>>>
>>>>>>> Thus, I will try to investigate the problem.
>>>>>>> Hopefully, I can change the buildbot script to get the build output
>>>>>>> directly on stdout instead of as html. The html output is
>>>>>>> currently not
>>>>>>> containing the corresponding information about the build of module
>>>>>>> apr.
>>>>>>>
>>>>>>
>>>>>> Done and Thx to Herbert triggering a clean build.
>>>>>> Unfortunately, the build was successful. Thus, I assume the reason
>>>>>> that we had no nightly windows builds from trunk since 2013-04-28
>>>>>> was
>>>>>> that no clean build had been performed.
>>>>> This is not the case. The clean build is not the panacea you see it
>>>>> as. As mentioned in several other communications, I went onto the
>>>>> box
>>>>> and cleaned up some processes that were hung (win7, win7snap, and
>>>>> win7ia2). All built successfully - even though the other two were
>>>>> incremental. The hung processes tend to occur /more/ during clean
>>>>> builds - not all the time, just more often. Thus, clean builds are
>>>>> more
>>>>> likely to create this type of build failure, they are not a fix as
>>>>> you're suggesting.
>>>>>
>>>>
>>>> Thanks for the information.
>>>> In order to have something more tangible for fixing this defect of
>>>> hanging build processes I propose to start an corresponding
>>>> investigation.
>>>> At least we should have a look after each build, esp. after each clean
>>>> build, if there are processes which hang.
>>>
>>> It seems that this defect just occured with build #105 of aoo-w7ia2 -
>>> see [1]. The build had been "killed". I assume that the one or the
>>> other process of this build is still working.
>>> Can somebody with corresponding karma check, if there are again
>>> hanging processes?
>>>
>>> [1] http://ci.apache.org/builders/aoo-w7ia2/builds/105
>> The build was not "killed" - the process that was running didn't report
>> back in 12000 sec = 200 min or 3+hours
>> At that point the buildbot tries to clean up, but this is the reaction,
>> not the root cause.
>>
>> command timed out: 12000 seconds without output, killing pid 2472
>> SIGKILL failed to kill process
>> using fake rc=-1
>> program finished with exit code -1
>>
>
> That is what I meant by "killed" - sorry for not expressing myself clear.
>
>> When I checked it later, the process was still hung (thus it's unlikely
>> that our problem is just with the length of the timeout). This is what
>> one of these hung processes looks like, and any subsequent builds will
>> fail if it's not cleaned up, as the processes lock files and block
>> subsequent compiles of the same package.
>>
>
> Thanks for having a look.
> Build #106 of aoo-w7ia2 went well after your clean up.
> Unfortunately, build #107 of aoo-w7ia2 had again the same failure as
> build #105.
ok - so I am now killing the hung processes - here is what I find:
* cl.exe - cl /nologo /?
* sh.exe - C:\cygwin\bin\sh.exe -c "dmake -P2 verbose=true >
/cygdrive/.../apr.txt 2>&1"
* perl.exe - C:\cygwin\bin\perl.exe E:/.../build.pl --all --html -P2
-- -P2
when I kill the cl process the other two come to life - i.e. that's the
hung process
looking in to the html progress page, the build is now finishing without
reporting errors.
A.
>
> Best regards, Oliver.
>
>>
>>>
>>>
>>> Best regards, Oliver.
>>>
>>>> Andrew, can only you perform such an investigation, because (as far
>>>> as I
>>>> know) you are the only who have direct access on the machine?
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 17.05.2013 01:53, Andrew Rist wrote:
>
>>>>>>
>>>>>> our buildbot building trunk nightly for windows has problems in modul
>>>>>> apr since a couple of days.
>>>>>> Unfortunately, the log does provide nothing for a reason not known to
>>>>>> me.
>>>>>>
>>>>>> Thus, I will try to investigate the problem.
>>>>>> Hopefully, I can change the buildbot script to get the build output
>>>>>> directly on stdout instead of as html. The html output is
>>>>>> currently not
>>>>>> containing the corresponding information about the build of module
>>>>>> apr.
>>>>>>
>>>>>
>>>>> Done and Thx to Herbert triggering a clean build.
>>>>> Unfortunately, the build was successful. Thus, I assume the reason
>>>>> that we had no nightly windows builds from trunk since 2013-04-28 was
>>>>> that no clean build had been performed.
>>>> This is not the case. The clean build is not the panacea you see it
>>>> as. As mentioned in several other communications, I went onto the box
>>>> and cleaned up some processes that were hung (win7, win7snap, and
>>>> win7ia2). All built successfully - even though the other two were
>>>> incremental. The hung processes tend to occur /more/ during clean
>>>> builds - not all the time, just more often. Thus, clean builds are
>>>> more
>>>> likely to create this type of build failure, they are not a fix as
>>>> you're suggesting.
>>>>
>>>
>>> Thanks for the information.
>>> In order to have something more tangible for fixing this defect of
>>> hanging build processes I propose to start an corresponding
>>> investigation.
>>> At least we should have a look after each build, esp. after each clean
>>> build, if there are processes which hang.
>>
>> It seems that this defect just occured with build #105 of aoo-w7ia2 -
>> see [1]. The build had been "killed". I assume that the one or the
>> other process of this build is still working.
>> Can somebody with corresponding karma check, if there are again
>> hanging processes?
>>
>> [1] http://ci.apache.org/builders/aoo-w7ia2/builds/105
> The build was not "killed" - the process that was running didn't report
> back in 12000 sec = 200 min or 3+hours
> At that point the buildbot tries to clean up, but this is the reaction,
> not the root cause.
>
> command timed out: 12000 seconds without output, killing pid 2472
> SIGKILL failed to kill process
> using fake rc=-1
> program finished with exit code -1
>
That is what I meant by "killed" - sorry for not expressing myself clear.
> When I checked it later, the process was still hung (thus it's unlikely
> that our problem is just with the length of the timeout). This is what
> one of these hung processes looks like, and any subsequent builds will
> fail if it's not cleaned up, as the processes lock files and block
> subsequent compiles of the same package.
>
Thanks for having a look.
Build #106 of aoo-w7ia2 went well after your clean up.
Unfortunately, build #107 of aoo-w7ia2 had again the same failure as
build #105.
Best regards, Oliver.
>
>>
>>
>> Best regards, Oliver.
>>
>>> Andrew, can only you perform such an investigation, because (as far as I
>>> know) you are the only who have direct access on the machine?
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Andrew Rist <an...@oracle.com>.
On 5/16/2013 1:31 AM, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 16.05.2013 10:24, Oliver-Rainer Wittmann wrote:
>> Hi,
>>
>> On 15.05.2013 21:51, Andrew Rist wrote:
>>>
>>> On 5/15/2013 12:03 AM, Oliver-Rainer Wittmann wrote:
>>>> Hi,
>>>>
>>>> On 14.05.2013 10:16, Oliver-Rainer Wittmann wrote:
>>>>> Hi,
>>>>>
>>>>> our buildbot building trunk nightly for windows has problems in modul
>>>>> apr since a couple of days.
>>>>> Unfortunately, the log does provide nothing for a reason not known to
>>>>> me.
>>>>>
>>>>> Thus, I will try to investigate the problem.
>>>>> Hopefully, I can change the buildbot script to get the build output
>>>>> directly on stdout instead of as html. The html output is
>>>>> currently not
>>>>> containing the corresponding information about the build of module
>>>>> apr.
>>>>>
>>>>
>>>> Done and Thx to Herbert triggering a clean build.
>>>> Unfortunately, the build was successful. Thus, I assume the reason
>>>> that we had no nightly windows builds from trunk since 2013-04-28 was
>>>> that no clean build had been performed.
>>> This is not the case. The clean build is not the panacea you see it
>>> as. As mentioned in several other communications, I went onto the box
>>> and cleaned up some processes that were hung (win7, win7snap, and
>>> win7ia2). All built successfully - even though the other two were
>>> incremental. The hung processes tend to occur /more/ during clean
>>> builds - not all the time, just more often. Thus, clean builds are
>>> more
>>> likely to create this type of build failure, they are not a fix as
>>> you're suggesting.
>>>
>>
>> Thanks for the information.
>> In order to have something more tangible for fixing this defect of
>> hanging build processes I propose to start an corresponding
>> investigation.
>> At least we should have a look after each build, esp. after each clean
>> build, if there are processes which hang.
>
> It seems that this defect just occured with build #105 of aoo-w7ia2 -
> see [1]. The build had been "killed". I assume that the one or the
> other process of this build is still working.
> Can somebody with corresponding karma check, if there are again
> hanging processes?
>
> [1] http://ci.apache.org/builders/aoo-w7ia2/builds/105
The build was not "killed" - the process that was running didn't report
back in 12000 sec = 200 min or 3+hours
At that point the buildbot tries to clean up, but this is the reaction,
not the root cause.
command timed out: 12000 seconds without output, killing pid 2472
SIGKILL failed to kill process
using fake rc=-1
program finished with exit code -1
When I checked it later, the process was still hung (thus it's unlikely
that our problem is just with the length of the timeout). This is what
one of these hung processes looks like, and any subsequent builds will
fail if it's not cleaned up, as the processes lock files and block
subsequent compiles of the same package.
>
>
> Best regards, Oliver.
>
>> Andrew, can only you perform such an investigation, because (as far as I
>> know) you are the only who have direct access on the machine?
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 16.05.2013 10:24, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 15.05.2013 21:51, Andrew Rist wrote:
>>
>> On 5/15/2013 12:03 AM, Oliver-Rainer Wittmann wrote:
>>> Hi,
>>>
>>> On 14.05.2013 10:16, Oliver-Rainer Wittmann wrote:
>>>> Hi,
>>>>
>>>> our buildbot building trunk nightly for windows has problems in modul
>>>> apr since a couple of days.
>>>> Unfortunately, the log does provide nothing for a reason not known to
>>>> me.
>>>>
>>>> Thus, I will try to investigate the problem.
>>>> Hopefully, I can change the buildbot script to get the build output
>>>> directly on stdout instead of as html. The html output is currently not
>>>> containing the corresponding information about the build of module apr.
>>>>
>>>
>>> Done and Thx to Herbert triggering a clean build.
>>> Unfortunately, the build was successful. Thus, I assume the reason
>>> that we had no nightly windows builds from trunk since 2013-04-28 was
>>> that no clean build had been performed.
>> This is not the case. The clean build is not the panacea you see it
>> as. As mentioned in several other communications, I went onto the box
>> and cleaned up some processes that were hung (win7, win7snap, and
>> win7ia2). All built successfully - even though the other two were
>> incremental. The hung processes tend to occur /more/ during clean
>> builds - not all the time, just more often. Thus, clean builds are more
>> likely to create this type of build failure, they are not a fix as
>> you're suggesting.
>>
>
> Thanks for the information.
> In order to have something more tangible for fixing this defect of
> hanging build processes I propose to start an corresponding investigation.
> At least we should have a look after each build, esp. after each clean
> build, if there are processes which hang.
It seems that this defect just occured with build #105 of aoo-w7ia2 -
see [1]. The build had been "killed". I assume that the one or the other
process of this build is still working.
Can somebody with corresponding karma check, if there are again hanging
processes?
[1] http://ci.apache.org/builders/aoo-w7ia2/builds/105
Best regards, Oliver.
> Andrew, can only you perform such an investigation, because (as far as I
> know) you are the only who have direct access on the machine?
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 15.05.2013 21:51, Andrew Rist wrote:
>
> On 5/15/2013 12:03 AM, Oliver-Rainer Wittmann wrote:
>> Hi,
>>
>> On 14.05.2013 10:16, Oliver-Rainer Wittmann wrote:
>>> Hi,
>>>
>>> our buildbot building trunk nightly for windows has problems in modul
>>> apr since a couple of days.
>>> Unfortunately, the log does provide nothing for a reason not known to
>>> me.
>>>
>>> Thus, I will try to investigate the problem.
>>> Hopefully, I can change the buildbot script to get the build output
>>> directly on stdout instead of as html. The html output is currently not
>>> containing the corresponding information about the build of module apr.
>>>
>>
>> Done and Thx to Herbert triggering a clean build.
>> Unfortunately, the build was successful. Thus, I assume the reason
>> that we had no nightly windows builds from trunk since 2013-04-28 was
>> that no clean build had been performed.
> This is not the case. The clean build is not the panacea you see it
> as. As mentioned in several other communications, I went onto the box
> and cleaned up some processes that were hung (win7, win7snap, and
> win7ia2). All built successfully - even though the other two were
> incremental. The hung processes tend to occur /more/ during clean
> builds - not all the time, just more often. Thus, clean builds are more
> likely to create this type of build failure, they are not a fix as
> you're suggesting.
>
Thanks for the information.
In order to have something more tangible for fixing this defect of
hanging build processes I propose to start an corresponding investigation.
At least we should have a look after each build, esp. after each clean
build, if there are processes which hang.
Andrew, can only you perform such an investigation, because (as far as I
know) you are the only who have direct access on the machine?
Best regards, Oliver.
>
> A.
>
> (and good catch on the ext_source logs...)
>>
>> I am now reverting my temporary changes for the buildbot aoo-win7.
>>
>> Best regards, Oliver.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: [buildbot] investigate nightly windows build
Posted by Andrew Rist <an...@oracle.com>.
On 5/15/2013 12:03 AM, Oliver-Rainer Wittmann wrote:
> Hi,
>
> On 14.05.2013 10:16, Oliver-Rainer Wittmann wrote:
>> Hi,
>>
>> our buildbot building trunk nightly for windows has problems in modul
>> apr since a couple of days.
>> Unfortunately, the log does provide nothing for a reason not known to
>> me.
>>
>> Thus, I will try to investigate the problem.
>> Hopefully, I can change the buildbot script to get the build output
>> directly on stdout instead of as html. The html output is currently not
>> containing the corresponding information about the build of module apr.
>>
>
> Done and Thx to Herbert triggering a clean build.
> Unfortunately, the build was successful. Thus, I assume the reason
> that we had no nightly windows builds from trunk since 2013-04-28 was
> that no clean build had been performed.
This is not the case. The clean build is not the panacea you see it
as. As mentioned in several other communications, I went onto the box
and cleaned up some processes that were hung (win7, win7snap, and
win7ia2). All built successfully - even though the other two were
incremental. The hung processes tend to occur /more/ during clean
builds - not all the time, just more often. Thus, clean builds are more
likely to create this type of build failure, they are not a fix as
you're suggesting.
A.
(and good catch on the ext_source logs...)
>
> I am now reverting my temporary changes for the buildbot aoo-win7.
>
> Best regards, Oliver.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
Re: [buildbot] investigate nightly windows build
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 14.05.2013 10:16, Oliver-Rainer Wittmann wrote:
> Hi,
>
> our buildbot building trunk nightly for windows has problems in modul
> apr since a couple of days.
> Unfortunately, the log does provide nothing for a reason not known to me.
>
> Thus, I will try to investigate the problem.
> Hopefully, I can change the buildbot script to get the build output
> directly on stdout instead of as html. The html output is currently not
> containing the corresponding information about the build of module apr.
>
Done and Thx to Herbert triggering a clean build.
Unfortunately, the build was successful. Thus, I assume the reason that
we had no nightly windows builds from trunk since 2013-04-28 was that no
clean build had been performed.
I am now reverting my temporary changes for the buildbot aoo-win7.
Best regards, Oliver.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org