You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Andre Fischer <aw...@gmail.com> on 2013/04/10 10:09:03 UTC
Build breaker and clean build
Hi,
tonight we had a build breaker in the windows build: a slot id that is
used in SW had been removed in SVX. The reference in SW had also been
removed, so this change should not be a problem.
But the windows build is still not a full build. Therefore the old SW
slot header files where used and the build broke.
There is an easy fix for situations like this: a clean build.
This may become important in the coming weeks when we have to fix some
bugs in the sidebar (which is about to be merged back into trunk). The
sidebar is implemented in several modules. Without a clean windows
build we will run into build breakers very regularly.
As I see it, we have a real problem here.
Regards,
Andre
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Build breaker and clean build
Posted by janI <ja...@apache.org>.
On 15 April 2013 14:00, Andre Fischer <aw...@gmail.com> wrote:
> I have updated the buildbot wiki page [1] to reflect all I know about it.
> Maybe those of you, who know more, can add your knowledge.
>
> One of the more important missing parts is how to trigger a clean build on
> Windows.
>
> -Andre
>
> [1] http://wiki.openoffice.org/**wiki/Buildbot<http://wiki.openoffice.org/wiki/Buildbot>
>
very informative, thanks. I think this is something we should reference in
the notes for newcomers.
rgds
jan I
>
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.org<de...@openoffice.apache.org>
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
Re: Build breaker and clean build
Posted by Andre Fischer <aw...@gmail.com>.
I have updated the buildbot wiki page [1] to reflect all I know about
it. Maybe those of you, who know more, can add your knowledge.
One of the more important missing parts is how to trigger a clean build
on Windows.
-Andre
[1] http://wiki.openoffice.org/wiki/Buildbot
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Build breaker and clean build
Posted by Herbert Dürr <hd...@apache.org>.
On 2013/04/10 10:27 AM, janI wrote:
>> I am no expert in what we can do in buildbot, but isnt it possible to run
> a script before build, that removes all unxlngx6.pro directories and
> solver, that should secure a clean build ??
Each nightly unxlngx6.pro buildbot run already does clean build. Only
the nightly windows build is incremental whereas the weekly windows
trunk and snapshot builds are done from scratch.
I suggested to default to a "better safe than sorry" approach of a clean
build but that was rejected at that time (2012/2) [1]
[1] http://markmail.org/message/wmlhc5f5zaiiyu2o
Herbert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Build breaker and clean build
Posted by janI <ja...@apache.org>.
On 10 April 2013 10:09, Andre Fischer <aw...@gmail.com> wrote:
> Hi,
>
> tonight we had a build breaker in the windows build: a slot id that is
> used in SW had been removed in SVX. The reference in SW had also been
> removed, so this change should not be a problem.
> But the windows build is still not a full build. Therefore the old SW
> slot header files where used and the build broke.
>
> There is an easy fix for situations like this: a clean build.
>
> This may become important in the coming weeks when we have to fix some
> bugs in the sidebar (which is about to be merged back into trunk). The
> sidebar is implemented in several modules. Without a clean windows build
> we will run into build breakers very regularly.
>
> As I see it, we have a real problem here.
>
> I am no expert in what we can do in buildbot, but isnt it possible to run
a script before build, that removes all unxlngx6.pro directories and
solver, that should secure a clean build ??
rgds
Jan I
> Regards,
> Andre
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.org<de...@openoffice.apache.org>
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
Re: Build breaker and clean build
Posted by Herbert Dürr <hd...@apache.org>.
On 2013/04/10 11:15 AM, Andre Fischer wrote:
> On 10.04.2013 10:33, Herbert Dürr wrote:
>> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>>> tonight we had a build breaker in the windows build: a slot id that is
>>> used in SW had been removed in SVX. The reference in SW had also been
>>> removed, so this change should not be a problem.
>>> But the windows build is still not a full build. Therefore the old SW
>>> slot header files where used and the build broke.
>>>
>>> There is an easy fix for situations like this: a clean build.
>>
>> Incremental build are known to have problems thats why I suggested [1]
>> to default to a clean build. That didn't receive consensus though and
>> indeed there are good reasons against it:
>>
>> The incremental build both tests the dependency system and it reduces
>> the load when building significantly.
>>
>> On the already strained buildbot this means a factor of almost five
>> improvement as clean build takes about 4.5h whereas an incremental
>> build takes only 0.5-1.0h.
>
> Hm, we prefer a fast build over a correct build?
Some developers apparently do. Please see the mail thread at [1]
[1] http://markmail.org/message/wmlhc5f5zaiiyu2o
> I don't think that that is the right priority. Also, a build that
> breaks due to errors caused by not starting clean is wasted time, ours
> and that of the build server.
I agree.
>> Andrew even had to reschedule the snapshot build away from the weekly
>> clean build because the buildbot load is a real problem.
>
> Would it help to do the windows build only every second day (night)?
Doing the regular build for trunk only every second night would only
help if the nightly builds for branches such as l10n or ia2 would also
be reduced. Do they still need nightly builds?
Herbert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Build breaker and clean build
Posted by Andre Fischer <aw...@gmail.com>.
On 10.04.2013 10:33, Herbert Dürr wrote:
> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>> tonight we had a build breaker in the windows build: a slot id that is
>> used in SW had been removed in SVX. The reference in SW had also been
>> removed, so this change should not be a problem.
>> But the windows build is still not a full build. Therefore the old SW
>> slot header files where used and the build broke.
>>
>> There is an easy fix for situations like this: a clean build.
>
> Incremental build are known to have problems thats why I suggested [1]
> to default to a clean build. That didn't receive consensus though and
> indeed there are good reasons against it:
>
> The incremental build both tests the dependency system and it reduces
> the load when building significantly.
>
> On the already strained buildbot this means a factor of almost five
> improvement as clean build takes about 4.5h whereas an incremental
> build takes only 0.5-1.0h.
Hm, we prefer a fast build over a correct build?
I don't think that that is the right priority. Also, a build that
breaks due to errors caused by not starting clean is wasted time, ours
and that of the build server.
>
> Andrew even had to reschedule the snapshot build away from the weekly
> clean build because the buildbot load is a real problem.
Would it help to do the windows build only every second day (night)?
-Andre
>
> [1] http://markmail.org/message/wmlhc5f5zaiiyu2o
> [2] http://markmail.org/message/7q64ijlwygdqmwf3
>
>> This may become important in the coming weeks when we have to fix some
>> bugs in the sidebar (which is about to be merged back into trunk). The
>> sidebar is implemented in several modules. Without a clean windows
>> build we will run into build breakers very regularly.
>
> It is possible to force a clean build manually.
>
> Herbert
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Build breaker and clean build
Posted by janI <ja...@apache.org>.
On Apr 11, 2013 6:43 PM, "Andrew Rist" <an...@oracle.com> wrote:
>
>
> On 4/11/2013 12:06 AM, Andre Fischer wrote:
>>
>> On 10.04.2013 18:57, Andrew Rist wrote:
>>>
>>>
>>> On 4/10/2013 1:33 AM, Herbert Dürr wrote:
>>>>
>>>> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>>>>>
>>>>> tonight we had a build breaker in the windows build: a slot id that is
>>>>> used in SW had been removed in SVX. The reference in SW had also been
>>>>> removed, so this change should not be a problem.
>>>>> But the windows build is still not a full build. Therefore the old SW
>>>>> slot header files where used and the build broke.
>>>>>
>>>>> There is an easy fix for situations like this: a clean build.
>>>>
>>>>
>>>> Incremental build are known to have problems thats why I suggested [1]
to default to a clean build. That didn't receive consensus though and
indeed there are good reasons against it:
>>>>
>>>> The incremental build both tests the dependency system and it reduces
the load when building significantly.
>>>>
>>>> On the already strained buildbot this means a factor of almost five
improvement as clean build takes about 4.5h whereas an incremental build
takes only 0.5-1.0h.
>>>>
>>>> Andrew even had to reschedule the snapshot build away from the weekly
clean build because the buildbot load is a real problem.
>>>>
>>>> [1] http://markmail.org/message/wmlhc5f5zaiiyu2o
>>>> [2] http://markmail.org/message/7q64ijlwygdqmwf3
>>>>
>>> Just to add here, that there are also issues with a clean build. The
clean build fails with some frequency on hung jobs and requires manual
attention.
>>
>>
>> That is one more reason to have more frequent clean builds so that we
can find the cause of these problems. They are not restricted to the build
bot. Others are affected, too. I would have tried to fix this, but I am
not able to reproduce this problem on my local machine.
>
>
> Sorry, I think I was unclear there. Due to the complexity of our build
process and the interaction with the buildbot, there is a reasonably high
incidence of false positive failures on clean builds. The Windows build
ends up with hung processes and throws an exception. If we were to switch
to clean builds we could expect several false positives a week, which would
require manual intervention. We have tests of clean builds daily on the
linux boxes, so in terms of coverage of the entire AOO buildbot setup, we
have full builds covered. I see the fact that some of our builds are
incremental and some are clean, as a feature, not as a shortcoming.
>
>
>>
>>> In reality, breaking changes that require a clean build are pretty
rare. For me, the clean build on the weekend and incremental during the
week seems to be a good compromise.
>>
>>
>> I am not sure about that. Besides, it is sometimes a bit difficult to
judge 'how incompatible' a change really is. Change a slot definition in
SVX and its use in SW. Do we need anything more? With a clean build we are
on the safe side.
>
> It's a trade off. How many false positives to you get, and how much
manual intervention does the system require. What I'm trying to
communicate is that my experience with 'this buildbot setup' suggests that
the current approach requires less of my time to keep healthy, and produces
less false positives.
>
>>
>> Besides, the clean-build-on-weekend policy would require us to hold all
incompatible changes until Friday, or live with a broken build during the
week.
>> I really thing that we need a better solution. A switch for marking a
change as incompatible and that would be interpreted by the build bot would
be the absolute minimum. But even that would call for trouble. At Sun we
have been there and it did not really work so well.
>
> This doesn't require waiting until the weekend, it requires a manual
clean run which can be kicked off easily. (I'm happy to show you, or hit
up Herbert - he has access, too) I don't disagree with your general
argument, I just see different trade-offs, and I consider this type failure
to have a fairly trivial recovery (kicking off a manual clean build).
just my little 2cent, andrew you are (in my opinion) the one who really
have hands on with our build system (apologize to everyone else who has
hands on, but I see andrew daily on #asftest).
Can you please state your recommendation clearly, and you have my support.
rgds
jan I
>
> Andrew
>
>>
>>>
>>>>> This may become important in the coming weeks when we have to fix some
>>>>> bugs in the sidebar (which is about to be merged back into trunk).
The
>>>>> sidebar is implemented in several modules. Without a clean windows
>>>>> build we will run into build breakers very regularly.
>>>>
>>>>
>>>> It is possible to force a clean build manually.
>>>
>>> I'm cleaning it up now and kicking off a build.
>>
>>
>> Thanks for taking care of this one.
>>
>> -Andre
>>
>>> Andrew
>>>
>>>
>>>>
>>>> Herbert
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
Re: Build breaker and clean build
Posted by janI <ja...@apache.org>.
On 12 April 2013 09:27, Andre Fischer <aw...@gmail.com> wrote:
> On 11.04.2013 18:43, Andrew Rist wrote:
>
>>
>> On 4/11/2013 12:06 AM, Andre Fischer wrote:
>>
>>> On 10.04.2013 18:57, Andrew Rist wrote:
>>>
>>>>
>>>> On 4/10/2013 1:33 AM, Herbert Dürr wrote:
>>>>
>>>>> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>>>>>
>>>>>> tonight we had a build breaker in the windows build: a slot id that is
>>>>>> used in SW had been removed in SVX. The reference in SW had also been
>>>>>> removed, so this change should not be a problem.
>>>>>> But the windows build is still not a full build. Therefore the old SW
>>>>>> slot header files where used and the build broke.
>>>>>>
>>>>>> There is an easy fix for situations like this: a clean build.
>>>>>>
>>>>>
>>>>> Incremental build are known to have problems thats why I suggested [1]
>>>>> to default to a clean build. That didn't receive consensus though and
>>>>> indeed there are good reasons against it:
>>>>>
>>>>> The incremental build both tests the dependency system and it reduces
>>>>> the load when building significantly.
>>>>>
>>>>> On the already strained buildbot this means a factor of almost five
>>>>> improvement as clean build takes about 4.5h whereas an incremental build
>>>>> takes only 0.5-1.0h.
>>>>>
>>>>> Andrew even had to reschedule the snapshot build away from the weekly
>>>>> clean build because the buildbot load is a real problem.
>>>>>
>>>>> [1] http://markmail.org/message/**wmlhc5f5zaiiyu2o<http://markmail.org/message/wmlhc5f5zaiiyu2o>
>>>>> [2] http://markmail.org/message/**7q64ijlwygdqmwf3<http://markmail.org/message/7q64ijlwygdqmwf3>
>>>>>
>>>>> Just to add here, that there are also issues with a clean build. The
>>>> clean build fails with some frequency on hung jobs and requires manual
>>>> attention.
>>>>
>>>
>>> That is one more reason to have more frequent clean builds so that we
>>> can find the cause of these problems. They are not restricted to the build
>>> bot. Others are affected, too. I would have tried to fix this, but I am
>>> not able to reproduce this problem on my local machine.
>>>
>>
>> Sorry, I think I was unclear there. Due to the complexity of our build
>> process and the interaction with the buildbot, there is a reasonably high
>> incidence of false positive failures on clean builds. The Windows build
>> ends up with hung processes and throws an exception. If we were to switch
>> to clean builds we could expect several false positives a week, which would
>> require manual intervention. We have tests of clean builds daily on the
>> linux boxes, so in terms of coverage of the entire AOO buildbot setup, we
>> have full builds covered. I see the fact that some of our builds are
>> incremental and some are clean, as a feature, not as a shortcoming.
>>
>
> OK, I understand. I see two problems with this approach:
>
> 1. We will not find the build problems when we continue to hide them with
> workarounds. Fixing them would benefit others as well, not just the build
> bots.
>
> 2. The task of the build bots is not only to verify that our source code
> is buildable but also to provide download sets for developers and QA.
>
>
>>
>>> In reality, breaking changes that require a clean build are pretty
>>>> rare. For me, the clean build on the weekend and incremental during the
>>>> week seems to be a good compromise.
>>>>
>>>
>>> I am not sure about that. Besides, it is sometimes a bit difficult to
>>> judge 'how incompatible' a change really is. Change a slot definition in
>>> SVX and its use in SW. Do we need anything more? With a clean build we are
>>> on the safe side.
>>>
>> It's a trade off. How many false positives to you get, and how much
>> manual intervention does the system require. What I'm trying to
>> communicate is that my experience with 'this buildbot setup' suggests that
>> the current approach requires less of my time to keep healthy, and produces
>> less false positives.
>>
>
> Ah, again I understand. It is a matter of pragmatism. I can accept that.
> I and I think all others on this list are thankful for your work in this
> area. And maybe we can help to improve the current state. Is there any
> documentation, a Wiki page maybe, to get an overview?
>
> Maybe we can rededicate one of the daily builds of one of the branches to
> debugging our windows build problems? Add a step to the build setup to
> apply patches that provide us with more debug info?
>
If I somehow can get my hands on the windows build bot vm, I can let it run
on my own server, and thereby free up the l10n build.
rgds
Jan I.
>
>
>
>>> Besides, the clean-build-on-weekend policy would require us to hold all
>>> incompatible changes until Friday, or live with a broken build during the
>>> week.
>>> I really thing that we need a better solution. A switch for marking a
>>> change as incompatible and that would be interpreted by the build bot would
>>> be the absolute minimum. But even that would call for trouble. At Sun we
>>> have been there and it did not really work so well.
>>>
>> This doesn't require waiting until the weekend, it requires a manual
>> clean run which can be kicked off easily. (I'm happy to show you, or hit
>> up Herbert - he has access, too)
>>
>
> It would be good to have this knowledge written down somewhere for easy
> reference. Especially if it really remains our only way to get clean
> builds.
>
> I don't disagree with your general argument, I just see different
>> trade-offs, and I consider this type failure to have a fairly trivial
>> recovery (kicking off a manual clean build).
>>
>
> Again, I don't think that this is a trivial solution.
>
> First I have to detect that my changes are incompatible. That is
> sometimes easy but sometimes it is not. If it is not easy then there is
> even the possibility that the incremental build will complete successful.
> But the result might show some very subtle errors that we may or may not
> find before a release.
> I would rather not spend any time on the analysis.
>
> Then we have to coordinate the triggering of incompatible builds. We have
> developers working around the world at every time of the day. It might be
> better to have an incompatible build to start a a fixed time then have
> developer A trigger a clean build and developer B, who is one hour late,
> has to do that again right after that. This might flood the build servers
> with AOO builds.
>
> -Andre
>
>
>> Andrew
>>
>>>
>>>
>>>> This may become important in the coming weeks when we have to fix some
>>>>>> bugs in the sidebar (which is about to be merged back into trunk).
>>>>>> The
>>>>>> sidebar is implemented in several modules. Without a clean windows
>>>>>> build we will run into build breakers very regularly.
>>>>>>
>>>>>
>>>>> It is possible to force a clean build manually.
>>>>>
>>>> I'm cleaning it up now and kicking off a build.
>>>>
>>>
>>> Thanks for taking care of this one.
>>>
>>> -Andre
>>>
>>> Andrew
>>>>
>>>>
>>>>
>>>>> Herbert
>>>>>
>>>>>
>>>>> ------------------------------**------------------------------**
>>>>> ---------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.org<de...@openoffice.apache.org>
>>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>>
>>>>>
>>>>
>>>> ------------------------------**------------------------------**
>>>> ---------
>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.org<de...@openoffice.apache.org>
>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>
>>>>
>>>
>>> ------------------------------**------------------------------**
>>> ---------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.org<de...@openoffice.apache.org>
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>>
>>
>> ------------------------------**------------------------------**---------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.org<de...@openoffice.apache.org>
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.org<de...@openoffice.apache.org>
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
Re: Build breaker and clean build
Posted by Andre Fischer <aw...@gmail.com>.
On 11.04.2013 18:43, Andrew Rist wrote:
>
> On 4/11/2013 12:06 AM, Andre Fischer wrote:
>> On 10.04.2013 18:57, Andrew Rist wrote:
>>>
>>> On 4/10/2013 1:33 AM, Herbert Dürr wrote:
>>>> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>>>>> tonight we had a build breaker in the windows build: a slot id
>>>>> that is
>>>>> used in SW had been removed in SVX. The reference in SW had also
>>>>> been
>>>>> removed, so this change should not be a problem.
>>>>> But the windows build is still not a full build. Therefore the old SW
>>>>> slot header files where used and the build broke.
>>>>>
>>>>> There is an easy fix for situations like this: a clean build.
>>>>
>>>> Incremental build are known to have problems thats why I suggested
>>>> [1] to default to a clean build. That didn't receive consensus
>>>> though and indeed there are good reasons against it:
>>>>
>>>> The incremental build both tests the dependency system and it
>>>> reduces the load when building significantly.
>>>>
>>>> On the already strained buildbot this means a factor of almost five
>>>> improvement as clean build takes about 4.5h whereas an incremental
>>>> build takes only 0.5-1.0h.
>>>>
>>>> Andrew even had to reschedule the snapshot build away from the
>>>> weekly clean build because the buildbot load is a real problem.
>>>>
>>>> [1] http://markmail.org/message/wmlhc5f5zaiiyu2o
>>>> [2] http://markmail.org/message/7q64ijlwygdqmwf3
>>>>
>>> Just to add here, that there are also issues with a clean build. The
>>> clean build fails with some frequency on hung jobs and requires
>>> manual attention.
>>
>> That is one more reason to have more frequent clean builds so that we
>> can find the cause of these problems. They are not restricted to the
>> build bot. Others are affected, too. I would have tried to fix
>> this, but I am not able to reproduce this problem on my local machine.
>
> Sorry, I think I was unclear there. Due to the complexity of our
> build process and the interaction with the buildbot, there is a
> reasonably high incidence of false positive failures on clean builds.
> The Windows build ends up with hung processes and throws an
> exception. If we were to switch to clean builds we could expect
> several false positives a week, which would require manual
> intervention. We have tests of clean builds daily on the linux boxes,
> so in terms of coverage of the entire AOO buildbot setup, we have full
> builds covered. I see the fact that some of our builds are
> incremental and some are clean, as a feature, not as a shortcoming.
OK, I understand. I see two problems with this approach:
1. We will not find the build problems when we continue to hide them
with workarounds. Fixing them would benefit others as well, not just
the build bots.
2. The task of the build bots is not only to verify that our source code
is buildable but also to provide download sets for developers and QA.
>
>>
>>> In reality, breaking changes that require a clean build are pretty
>>> rare. For me, the clean build on the weekend and incremental during
>>> the week seems to be a good compromise.
>>
>> I am not sure about that. Besides, it is sometimes a bit difficult
>> to judge 'how incompatible' a change really is. Change a slot
>> definition in SVX and its use in SW. Do we need anything more? With
>> a clean build we are on the safe side.
> It's a trade off. How many false positives to you get, and how much
> manual intervention does the system require. What I'm trying to
> communicate is that my experience with 'this buildbot setup' suggests
> that the current approach requires less of my time to keep healthy,
> and produces less false positives.
Ah, again I understand. It is a matter of pragmatism. I can accept
that. I and I think all others on this list are thankful for your work
in this area. And maybe we can help to improve the current state. Is
there any documentation, a Wiki page maybe, to get an overview?
Maybe we can rededicate one of the daily builds of one of the branches
to debugging our windows build problems? Add a step to the build setup
to apply patches that provide us with more debug info?
>>
>> Besides, the clean-build-on-weekend policy would require us to hold
>> all incompatible changes until Friday, or live with a broken build
>> during the week.
>> I really thing that we need a better solution. A switch for marking
>> a change as incompatible and that would be interpreted by the build
>> bot would be the absolute minimum. But even that would call for
>> trouble. At Sun we have been there and it did not really work so well.
> This doesn't require waiting until the weekend, it requires a manual
> clean run which can be kicked off easily. (I'm happy to show you, or
> hit up Herbert - he has access, too)
It would be good to have this knowledge written down somewhere for easy
reference. Especially if it really remains our only way to get clean
builds.
> I don't disagree with your general argument, I just see different
> trade-offs, and I consider this type failure to have a fairly trivial
> recovery (kicking off a manual clean build).
Again, I don't think that this is a trivial solution.
First I have to detect that my changes are incompatible. That is
sometimes easy but sometimes it is not. If it is not easy then there is
even the possibility that the incremental build will complete
successful. But the result might show some very subtle errors that we
may or may not find before a release.
I would rather not spend any time on the analysis.
Then we have to coordinate the triggering of incompatible builds. We
have developers working around the world at every time of the day. It
might be better to have an incompatible build to start a a fixed time
then have developer A trigger a clean build and developer B, who is one
hour late, has to do that again right after that. This might flood the
build servers with AOO builds.
-Andre
>
> Andrew
>>
>>>
>>>>> This may become important in the coming weeks when we have to fix
>>>>> some
>>>>> bugs in the sidebar (which is about to be merged back into
>>>>> trunk). The
>>>>> sidebar is implemented in several modules. Without a clean windows
>>>>> build we will run into build breakers very regularly.
>>>>
>>>> It is possible to force a clean build manually.
>>> I'm cleaning it up now and kicking off a build.
>>
>> Thanks for taking care of this one.
>>
>> -Andre
>>
>>> Andrew
>>>
>>>
>>>>
>>>> Herbert
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Build breaker and clean build
Posted by Andrew Rist <an...@oracle.com>.
On 4/11/2013 12:06 AM, Andre Fischer wrote:
> On 10.04.2013 18:57, Andrew Rist wrote:
>>
>> On 4/10/2013 1:33 AM, Herbert Dürr wrote:
>>> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>>>> tonight we had a build breaker in the windows build: a slot id that is
>>>> used in SW had been removed in SVX. The reference in SW had also been
>>>> removed, so this change should not be a problem.
>>>> But the windows build is still not a full build. Therefore the old SW
>>>> slot header files where used and the build broke.
>>>>
>>>> There is an easy fix for situations like this: a clean build.
>>>
>>> Incremental build are known to have problems thats why I suggested
>>> [1] to default to a clean build. That didn't receive consensus
>>> though and indeed there are good reasons against it:
>>>
>>> The incremental build both tests the dependency system and it
>>> reduces the load when building significantly.
>>>
>>> On the already strained buildbot this means a factor of almost five
>>> improvement as clean build takes about 4.5h whereas an incremental
>>> build takes only 0.5-1.0h.
>>>
>>> Andrew even had to reschedule the snapshot build away from the
>>> weekly clean build because the buildbot load is a real problem.
>>>
>>> [1] http://markmail.org/message/wmlhc5f5zaiiyu2o
>>> [2] http://markmail.org/message/7q64ijlwygdqmwf3
>>>
>> Just to add here, that there are also issues with a clean build. The
>> clean build fails with some frequency on hung jobs and requires
>> manual attention.
>
> That is one more reason to have more frequent clean builds so that we
> can find the cause of these problems. They are not restricted to the
> build bot. Others are affected, too. I would have tried to fix
> this, but I am not able to reproduce this problem on my local machine.
Sorry, I think I was unclear there. Due to the complexity of our build
process and the interaction with the buildbot, there is a reasonably
high incidence of false positive failures on clean builds. The Windows
build ends up with hung processes and throws an exception. If we were
to switch to clean builds we could expect several false positives a
week, which would require manual intervention. We have tests of clean
builds daily on the linux boxes, so in terms of coverage of the entire
AOO buildbot setup, we have full builds covered. I see the fact that
some of our builds are incremental and some are clean, as a feature, not
as a shortcoming.
>
>> In reality, breaking changes that require a clean build are pretty
>> rare. For me, the clean build on the weekend and incremental during
>> the week seems to be a good compromise.
>
> I am not sure about that. Besides, it is sometimes a bit difficult
> to judge 'how incompatible' a change really is. Change a slot
> definition in SVX and its use in SW. Do we need anything more? With a
> clean build we are on the safe side.
It's a trade off. How many false positives to you get, and how much
manual intervention does the system require. What I'm trying to
communicate is that my experience with 'this buildbot setup' suggests
that the current approach requires less of my time to keep healthy, and
produces less false positives.
>
> Besides, the clean-build-on-weekend policy would require us to hold
> all incompatible changes until Friday, or live with a broken build
> during the week.
> I really thing that we need a better solution. A switch for marking a
> change as incompatible and that would be interpreted by the build bot
> would be the absolute minimum. But even that would call for trouble.
> At Sun we have been there and it did not really work so well.
This doesn't require waiting until the weekend, it requires a manual
clean run which can be kicked off easily. (I'm happy to show you, or
hit up Herbert - he has access, too) I don't disagree with your
general argument, I just see different trade-offs, and I consider this
type failure to have a fairly trivial recovery (kicking off a manual
clean build).
Andrew
>
>>
>>>> This may become important in the coming weeks when we have to fix some
>>>> bugs in the sidebar (which is about to be merged back into trunk).
>>>> The
>>>> sidebar is implemented in several modules. Without a clean windows
>>>> build we will run into build breakers very regularly.
>>>
>>> It is possible to force a clean build manually.
>> I'm cleaning it up now and kicking off a build.
>
> Thanks for taking care of this one.
>
> -Andre
>
>> Andrew
>>
>>
>>>
>>> Herbert
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Build breaker and clean build
Posted by Andre Fischer <aw...@gmail.com>.
On 10.04.2013 18:57, Andrew Rist wrote:
>
> On 4/10/2013 1:33 AM, Herbert Dürr wrote:
>> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>>> tonight we had a build breaker in the windows build: a slot id that is
>>> used in SW had been removed in SVX. The reference in SW had also been
>>> removed, so this change should not be a problem.
>>> But the windows build is still not a full build. Therefore the old SW
>>> slot header files where used and the build broke.
>>>
>>> There is an easy fix for situations like this: a clean build.
>>
>> Incremental build are known to have problems thats why I suggested
>> [1] to default to a clean build. That didn't receive consensus though
>> and indeed there are good reasons against it:
>>
>> The incremental build both tests the dependency system and it reduces
>> the load when building significantly.
>>
>> On the already strained buildbot this means a factor of almost five
>> improvement as clean build takes about 4.5h whereas an incremental
>> build takes only 0.5-1.0h.
>>
>> Andrew even had to reschedule the snapshot build away from the weekly
>> clean build because the buildbot load is a real problem.
>>
>> [1] http://markmail.org/message/wmlhc5f5zaiiyu2o
>> [2] http://markmail.org/message/7q64ijlwygdqmwf3
>>
> Just to add here, that there are also issues with a clean build. The
> clean build fails with some frequency on hung jobs and requires manual
> attention.
That is one more reason to have more frequent clean builds so that we
can find the cause of these problems. They are not restricted to the
build bot. Others are affected, too. I would have tried to fix this,
but I am not able to reproduce this problem on my local machine.
> In reality, breaking changes that require a clean build are pretty
> rare. For me, the clean build on the weekend and incremental during
> the week seems to be a good compromise.
I am not sure about that. Besides, it is sometimes a bit difficult to
judge 'how incompatible' a change really is. Change a slot definition
in SVX and its use in SW. Do we need anything more? With a clean build
we are on the safe side.
Besides, the clean-build-on-weekend policy would require us to hold all
incompatible changes until Friday, or live with a broken build during
the week.
I really thing that we need a better solution. A switch for marking a
change as incompatible and that would be interpreted by the build bot
would be the absolute minimum. But even that would call for trouble.
At Sun we have been there and it did not really work so well.
>
>>> This may become important in the coming weeks when we have to fix some
>>> bugs in the sidebar (which is about to be merged back into trunk). The
>>> sidebar is implemented in several modules. Without a clean windows
>>> build we will run into build breakers very regularly.
>>
>> It is possible to force a clean build manually.
> I'm cleaning it up now and kicking off a build.
Thanks for taking care of this one.
-Andre
> Andrew
>
>
>>
>> Herbert
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Build breaker and clean build
Posted by janI <ja...@apache.org>.
On Apr 10, 2013 6:57 PM, "Andrew Rist" <an...@oracle.com> wrote:
>
>
> On 4/10/2013 1:33 AM, Herbert Dürr wrote:
>>
>> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>>>
>>> tonight we had a build breaker in the windows build: a slot id that is
>>> used in SW had been removed in SVX. The reference in SW had also been
>>> removed, so this change should not be a problem.
>>> But the windows build is still not a full build. Therefore the old SW
>>> slot header files where used and the build broke.
>>>
>>> There is an easy fix for situations like this: a clean build.
>>
>>
>> Incremental build are known to have problems thats why I suggested [1]
to default to a clean build. That didn't receive consensus though and
indeed there are good reasons against it:
>>
>> The incremental build both tests the dependency system and it reduces
the load when building significantly.
>>
>> On the already strained buildbot this means a factor of almost five
improvement as clean build takes about 4.5h whereas an incremental build
takes only 0.5-1.0h.
>>
>> Andrew even had to reschedule the snapshot build away from the weekly
clean build because the buildbot load is a real problem.
>>
>> [1] http://markmail.org/message/wmlhc5f5zaiiyu2o
>> [2] http://markmail.org/message/7q64ijlwygdqmwf3
>>
> Just to add here, that there are also issues with a clean build. The
clean build fails with some frequency on hung jobs and requires manual
attention.
> In reality, breaking changes that require a clean build are pretty rare.
For me, the clean build on the weekend and incremental during the week
seems to be a good compromise.
who whispered "new build system" :-)
+1 please consider it also for l10n, I will shortly start making many
makefile changes.
rgd
jan I
>
>
>>> This may become important in the coming weeks when we have to fix some
>>> bugs in the sidebar (which is about to be merged back into trunk). The
>>> sidebar is implemented in several modules. Without a clean windows
>>> build we will run into build breakers very regularly.
>>
>>
>> It is possible to force a clean build manually.
>
> I'm cleaning it up now and kicking off a build.
> Andrew
>
>
>
>>
>> Herbert
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
Re: Build breaker and clean build
Posted by Andrew Rist <an...@oracle.com>.
On 4/10/2013 1:33 AM, Herbert Dürr wrote:
> On 2013/04/10 10:09 AM, Andre Fischer wrote:
>> tonight we had a build breaker in the windows build: a slot id that is
>> used in SW had been removed in SVX. The reference in SW had also been
>> removed, so this change should not be a problem.
>> But the windows build is still not a full build. Therefore the old SW
>> slot header files where used and the build broke.
>>
>> There is an easy fix for situations like this: a clean build.
>
> Incremental build are known to have problems thats why I suggested [1]
> to default to a clean build. That didn't receive consensus though and
> indeed there are good reasons against it:
>
> The incremental build both tests the dependency system and it reduces
> the load when building significantly.
>
> On the already strained buildbot this means a factor of almost five
> improvement as clean build takes about 4.5h whereas an incremental
> build takes only 0.5-1.0h.
>
> Andrew even had to reschedule the snapshot build away from the weekly
> clean build because the buildbot load is a real problem.
>
> [1] http://markmail.org/message/wmlhc5f5zaiiyu2o
> [2] http://markmail.org/message/7q64ijlwygdqmwf3
>
Just to add here, that there are also issues with a clean build. The
clean build fails with some frequency on hung jobs and requires manual
attention.
In reality, breaking changes that require a clean build are pretty
rare. For me, the clean build on the weekend and incremental during the
week seems to be a good compromise.
>> This may become important in the coming weeks when we have to fix some
>> bugs in the sidebar (which is about to be merged back into trunk). The
>> sidebar is implemented in several modules. Without a clean windows
>> build we will run into build breakers very regularly.
>
> It is possible to force a clean build manually.
I'm cleaning it up now and kicking off a build.
Andrew
>
> Herbert
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Build breaker and clean build
Posted by Herbert Dürr <hd...@apache.org>.
On 2013/04/10 10:09 AM, Andre Fischer wrote:
> tonight we had a build breaker in the windows build: a slot id that is
> used in SW had been removed in SVX. The reference in SW had also been
> removed, so this change should not be a problem.
> But the windows build is still not a full build. Therefore the old SW
> slot header files where used and the build broke.
>
> There is an easy fix for situations like this: a clean build.
Incremental build are known to have problems thats why I suggested [1]
to default to a clean build. That didn't receive consensus though and
indeed there are good reasons against it:
The incremental build both tests the dependency system and it reduces
the load when building significantly.
On the already strained buildbot this means a factor of almost five
improvement as clean build takes about 4.5h whereas an incremental build
takes only 0.5-1.0h.
Andrew even had to reschedule the snapshot build away from the weekly
clean build because the buildbot load is a real problem.
[1] http://markmail.org/message/wmlhc5f5zaiiyu2o
[2] http://markmail.org/message/7q64ijlwygdqmwf3
> This may become important in the coming weeks when we have to fix some
> bugs in the sidebar (which is about to be merged back into trunk). The
> sidebar is implemented in several modules. Without a clean windows
> build we will run into build breakers very regularly.
It is possible to force a clean build manually.
Herbert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org