Hudson administrivia, build timeout

[Justin Mason <jm...@jmason.org>]

I'm installing the following plugins:

- Audit Trail plugin ('Keep a log of who performed particular Hudson
  operations, such as configuring jobs', handy in our configuration with
  so many users)

- Bugzilla Plugin ('This plugin integrates Bugzilla into Hudson', we use
  bugzilla in SpamAssassin and I'm sure there are others)

- Warnings Plugin ('This plugin generates the trend report for compiler
  warnings in the build log', looks pretty nifty!)

- and I'm going to re-try the Build Timeout plugin ('This plugin allows
  you to automatically abort a build if it's taking too long').
  
We tried the build timeout before, I think, and it didn't help.  But I
think some of the timeouts we're seeing now are due to broken tests on
some projects, and we've upgraded Hudson itself since the last try, so
it's worth a retry in my opinion.

I also checked the recent Hudson changelog, but nothing relevant has been
implemented that would fix build hangs.

Anyway, if your project has had problems with build hangs, please enable a
timeout on the "Configure" page.  It's about halfway down, in the "Build
Environment" section -- tick the '[x] Abort the build if it's stuck'
tickbox and set 'Timeout minutes' to a sane upper limit.  

If we run into builds of your projects timing out, we'll set this for you. ;)

--j.

Re: Hudson administrivia, build timeout

[Nigel Daley <ni...@apache.org>]

On Jul 10, 2009, at 4:37 AM, Justin Mason wrote:

> On Thu, Jul 9, 2009 at 14:12, Justin Mason<jm...@jmason.org> wrote:
>> On Thu, Jul 9, 2009 at 13:29, Jukka  
>> Zitting<ju...@gmail.com> wrote:
>>> Hi,
>>>
>>> On Thu, Jul 9, 2009 at 2:19 PM, Justin Mason<jm...@jmason.org> wrote:
>>>> FWIW, my experience over the last few days of monitoring has been  
>>>> that
>>>> our build backlogs on the Hudson machine are due to contention  
>>>> for the
>>>> limited number of executors; particularly the 2 on the main  
>>>> instance.
>>>> There are a few projects that perform 1.5-hour deployments from  
>>>> this.
>>>> IMO we need to come up with a way to accomodate this.
>>>
>>> Should we add a "long build" lock that all builds that normally take
>>> more than say 60 minutes should synchronize on? That way we'd never
>>> have situations where two long builds block both executors at the  
>>> same
>>> time.
>>
>> hmm, that's a good idea.  WDYT, Nigel?
>
> Nigel's on holidays. I've gone ahead and done this anyway ;)
>
> There are now two Locks: "Long-running jobs on
> hudson.zones.apache.org" and "Long-running jobs on
> minerva.apache.org".  (I haven't created one for the Lucene/Hadoop
> hosts.)
>
> Any builds that seem to be taking a very long time (fsvo "very long")
> will be changed to synch on those locks, in order to leave one of the
> executors free on those hosts for shorter builds.
>
> I've changed all the builds that seem to be consistently running for
> longer than 1 hour to sync on those locks.

Yes, great idea.  Thanks Justin!

Nige

Re: Hudson administrivia, build timeout

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Fri, Jul 10, 2009 at 1:37 PM, Justin Mason<jm...@jmason.org> wrote:
> Nigel's on holidays. I've gone ahead and done this anyway ;)

Excellent, thanks!

BR,

Jukka Zitting

Re: Hudson administrivia, build timeout

[Justin Mason <jm...@jmason.org>]

On Thu, Jul 9, 2009 at 14:12, Justin Mason<jm...@jmason.org> wrote:
> On Thu, Jul 9, 2009 at 13:29, Jukka Zitting<ju...@gmail.com> wrote:
>> Hi,
>>
>> On Thu, Jul 9, 2009 at 2:19 PM, Justin Mason<jm...@jmason.org> wrote:
>>> FWIW, my experience over the last few days of monitoring has been that
>>> our build backlogs on the Hudson machine are due to contention for the
>>> limited number of executors; particularly the 2 on the main instance.
>>> There are a few projects that perform 1.5-hour deployments from this.
>>> IMO we need to come up with a way to accomodate this.
>>
>> Should we add a "long build" lock that all builds that normally take
>> more than say 60 minutes should synchronize on? That way we'd never
>> have situations where two long builds block both executors at the same
>> time.
>
> hmm, that's a good idea.  WDYT, Nigel?

Nigel's on holidays. I've gone ahead and done this anyway ;)

There are now two Locks: "Long-running jobs on
hudson.zones.apache.org" and "Long-running jobs on
minerva.apache.org".  (I haven't created one for the Lucene/Hadoop
hosts.)

Any builds that seem to be taking a very long time (fsvo "very long")
will be changed to synch on those locks, in order to leave one of the
executors free on those hosts for shorter builds.

I've changed all the builds that seem to be consistently running for
longer than 1 hour to sync on those locks.

--j.

Re: Hudson administrivia, build timeout

[Justin Mason <jm...@jmason.org>]

On Thu, Jul 9, 2009 at 13:29, Jukka Zitting<ju...@gmail.com> wrote:
> Hi,
>
> On Thu, Jul 9, 2009 at 2:19 PM, Justin Mason<jm...@jmason.org> wrote:
>> FWIW, my experience over the last few days of monitoring has been that
>> our build backlogs on the Hudson machine are due to contention for the
>> limited number of executors; particularly the 2 on the main instance.
>> There are a few projects that perform 1.5-hour deployments from this.
>> IMO we need to come up with a way to accomodate this.
>
> Should we add a "long build" lock that all builds that normally take
> more than say 60 minutes should synchronize on? That way we'd never
> have situations where two long builds block both executors at the same
> time.

hmm, that's a good idea.  WDYT, Nigel?

Re: Hudson administrivia, build timeout

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Thu, Jul 9, 2009 at 2:19 PM, Justin Mason<jm...@jmason.org> wrote:
> FWIW, my experience over the last few days of monitoring has been that
> our build backlogs on the Hudson machine are due to contention for the
> limited number of executors; particularly the 2 on the main instance.
> There are a few projects that perform 1.5-hour deployments from this.
> IMO we need to come up with a way to accomodate this.

Should we add a "long build" lock that all builds that normally take
more than say 60 minutes should synchronize on? That way we'd never
have situations where two long builds block both executors at the same
time.

BR,

Jukka Zitting

Re: Hudson administrivia, build timeout

[Justin Mason <jm...@jmason.org>]

On Wed, Jul 8, 2009 at 21:43, Nigel Daley<nd...@yahoo-inc.com> wrote:
> On Jul 8, 2009, at 4:24 AM, Jukka Zitting wrote
>> On Wed, Jul 8, 2009 at 12:50 PM, Justin Mason<jm...@jmason.org> wrote:
>>>
>>> Should we set policy for the ASF hudson instance regarding the max
>>> runtime of builds, seeing as we only have 4 build executor slots?
>>
>> That would be good, though we may want to allow longer builds that
>> only run relatively seldom (e.g. weekly).
>
> +1.  There may need to be exceptions to this.  If I recall, the Harmony
> builds take a long time.

+1

OK, I suggest we set a relaxed policy of a 2-hour built timeout for
most projects.

If you see a project causing a backlog due to a longer build duration,
then feel free to log in and click through into its Configuration
page, where the build timeout can be set to 120 minutes. (and stop the
build, of course, using the [x].)

If the project indicates in its name, description, or build schedule
that it's a long-running, infrequent (e.g. weekly) build, then we can
relax that 2-hour limit and maybe allow it to run for longer, 4
hours/240 mins maybe?  Play it by ear, anyway. ;)

FWIW, my experience over the last few days of monitoring has been that
our build backlogs on the Hudson machine are due to contention for the
limited number of executors; particularly the 2 on the main instance.
There are a few projects that perform 1.5-hour deployments from this.
IMO we need to come up with a way to accomodate this.

I haven't observed any "hung" builds that the build timeout didn't
deal with, btw.

--j.

Re: Hudson administrivia, build timeout

[Nigel Daley <nd...@yahoo-inc.com>]

On Jul 8, 2009, at 4:24 AM, Jukka Zitting wrote:

> Hi,
>
> On Wed, Jul 8, 2009 at 12:50 PM, Justin Mason<jm...@jmason.org> wrote:
>> Should we set policy for the ASF hudson instance regarding the max
>> runtime of builds, seeing as we only have 4 build executor slots?
>
> That would be good, though we may want to allow longer builds that
> only run relatively seldom (e.g. weekly).

+1.  There may need to be exceptions to this.  If I recall, the  
Harmony builds take a long time.

Nige

Re: Hudson administrivia, build timeout

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Jul 8, 2009 at 12:50 PM, Justin Mason<jm...@jmason.org> wrote:
> Should we set policy for the ASF hudson instance regarding the max
> runtime of builds, seeing as we only have 4 build executor slots?

That would be good, though we may want to allow longer builds that
only run relatively seldom (e.g. weekly).

BR,

Jukka Zitting

Re: Hudson administrivia, build timeout

[Justin Mason <jm...@jmason.org>]

> We tried the build timeout before, I think, and it didn't help.  But I
> think some of the timeouts we're seeing now are due to broken tests on
> some projects, and we've upgraded Hudson itself since the last try, so
> it's worth a retry in my opinion.
>
> I also checked the recent Hudson changelog, but nothing relevant has been
> implemented that would fix build hangs.
>
> Anyway, if your project has had problems with build hangs, please enable a
> timeout on the "Configure" page.  It's about halfway down, in the "Build
> Environment" section -- tick the '[x] Abort the build if it's stuck'
> tickbox and set 'Timeout minutes' to a sane upper limit.
>
> If we run into builds of your projects timing out, we'll set this for you. ;)

update to this: so far all of the hangs (or at least very long-running
builds) since then have been Maven-related.  Is it expected that these
would take 2 hours to run?

Should we set policy for the ASF hudson instance regarding the max
runtime of builds, seeing as we only have 4 build executor slots?

--j.

Re: Hudson administrivia, build timeout

[Justin Mason <jm...@jmason.org>]

ok.  I think this is now fixed; I've made it require a "bug[space]" prefix.
annoyingly it's a global setting, not per-project :(

--j.

On Wed, Jul 8, 2009 at 21:45, Nigel Daley<nd...@yahoo-inc.com> wrote:
> Many thanks Justin!
>
> I think there's something funky with the Bugzilla Plugin config.
>
> Our Jira issues are being broken into 2 links now.  Example, MAPREDUCE-693
> is broken into
> 1) MAPREDUCE- links to Jira issue
> 2) 693 links to bugzilla
>
> Nige
>
> On Jul 7, 2009, at 3:43 AM, Justin Mason wrote:
>
>> I'm installing the following plugins:
>>
>> - Audit Trail plugin ('Keep a log of who performed particular Hudson
>>  operations, such as configuring jobs', handy in our configuration with
>>  so many users)
>>
>> - Bugzilla Plugin ('This plugin integrates Bugzilla into Hudson', we use
>>  bugzilla in SpamAssassin and I'm sure there are others)
>>
>> - Warnings Plugin ('This plugin generates the trend report for compiler
>>  warnings in the build log', looks pretty nifty!)
>>
>> - and I'm going to re-try the Build Timeout plugin ('This plugin allows
>>  you to automatically abort a build if it's taking too long').
>>
>> We tried the build timeout before, I think, and it didn't help.  But I
>> think some of the timeouts we're seeing now are due to broken tests on
>> some projects, and we've upgraded Hudson itself since the last try, so
>> it's worth a retry in my opinion.
>>
>> I also checked the recent Hudson changelog, but nothing relevant has been
>> implemented that would fix build hangs.
>>
>> Anyway, if your project has had problems with build hangs, please enable a
>> timeout on the "Configure" page.  It's about halfway down, in the "Build
>> Environment" section -- tick the '[x] Abort the build if it's stuck'
>> tickbox and set 'Timeout minutes' to a sane upper limit.
>>
>> If we run into builds of your projects timing out, we'll set this for you.
>> ;)
>>
>> --j.
>
>

Re: Hudson administrivia, build timeout

[Nigel Daley <nd...@yahoo-inc.com>]

Many thanks Justin!

I think there's something funky with the Bugzilla Plugin config.

Our Jira issues are being broken into 2 links now.  Example,  
MAPREDUCE-693 is broken into
1) MAPREDUCE- links to Jira issue
2) 693 links to bugzilla

Nige

On Jul 7, 2009, at 3:43 AM, Justin Mason wrote:

> I'm installing the following plugins:
>
> - Audit Trail plugin ('Keep a log of who performed particular Hudson
>  operations, such as configuring jobs', handy in our configuration  
> with
>  so many users)
>
> - Bugzilla Plugin ('This plugin integrates Bugzilla into Hudson', we  
> use
>  bugzilla in SpamAssassin and I'm sure there are others)
>
> - Warnings Plugin ('This plugin generates the trend report for  
> compiler
>  warnings in the build log', looks pretty nifty!)
>
> - and I'm going to re-try the Build Timeout plugin ('This plugin  
> allows
>  you to automatically abort a build if it's taking too long').
>
> We tried the build timeout before, I think, and it didn't help.  But I
> think some of the timeouts we're seeing now are due to broken tests on
> some projects, and we've upgraded Hudson itself since the last try, so
> it's worth a retry in my opinion.
>
> I also checked the recent Hudson changelog, but nothing relevant has  
> been
> implemented that would fix build hangs.
>
> Anyway, if your project has had problems with build hangs, please  
> enable a
> timeout on the "Configure" page.  It's about halfway down, in the  
> "Build
> Environment" section -- tick the '[x] Abort the build if it's stuck'
> tickbox and set 'Timeout minutes' to a sane upper limit.
>
> If we run into builds of your projects timing out, we'll set this  
> for you. ;)
>
> --j.