You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Andrew Bayer <an...@gmail.com> on 2015/08/20 19:32:51 UTC
FYI - disabled timer triggers across the board on builds.apache.org
I noticed a lot of jobs that were running every day regardless of whether
anything changed and eating up a lot of executors, so I bulk-removed all of
them, changing those jobs to poll for changes hourly instead. If your
project has one or two jobs that you need to run daily whether there are
code changes or not, you can re-enable the timer, but please do not do that
for more than a couple jobs, and please do not do it for jobs that take
longer than half an hour.
A.
Re: FYI - disabled timer triggers across the board on builds.apache.org
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
It turned out that I’ve incorrectly put the “run every 10 minutes” to “poll SCM” field rather then to “Run periodically” field.
Fixed.
Jarcec
> On Aug 21, 2015, at 9:29 AM, Jarek Jarcec Cecho <ja...@apache.org> wrote:
>
> No worries, I completely get why we made this change across the board. I actually feel bad that I didn’t realize the consequences yesterday when I read the email and had to spend good half an hour figuring it out :)
>
> Jarcec
>
>> On Aug 21, 2015, at 9:17 AM, Andrew Bayer <an...@gmail.com> wrote:
>>
>> Oops, sorry about that!
>> On Aug 21, 2015 12:02, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
>>
>>> I’ve noticed that this change has pretty much disabled Hadoop world
>>> spefici precommit [1] hook as this job is running every 10 minutes and
>>> fetching updates from JIRA, rather then following any repository. I’ve put
>>> back the “run every 10 minutes”. This job is usually done in less then 30
>>> seconds, so I’m assuming that it won’t be a problem :)
>>>
>>> Jarcec
>>>
>>> Links:
>>> 1: https://builds.apache.org/job/PreCommit-Admin
>>>
>>>> On Aug 20, 2015, at 10:32 AM, Andrew Bayer <an...@gmail.com>
>>> wrote:
>>>>
>>>> I noticed a lot of jobs that were running every day regardless of whether
>>>> anything changed and eating up a lot of executors, so I bulk-removed all
>>> of
>>>> them, changing those jobs to poll for changes hourly instead. If your
>>>> project has one or two jobs that you need to run daily whether there are
>>>> code changes or not, you can re-enable the timer, but please do not do
>>> that
>>>> for more than a couple jobs, and please do not do it for jobs that take
>>>> longer than half an hour.
>>>>
>>>> A.
>>>
>>>
>
Re: FYI - disabled timer triggers across the board on builds.apache.org
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
No worries, I completely get why we made this change across the board. I actually feel bad that I didn’t realize the consequences yesterday when I read the email and had to spend good half an hour figuring it out :)
Jarcec
> On Aug 21, 2015, at 9:17 AM, Andrew Bayer <an...@gmail.com> wrote:
>
> Oops, sorry about that!
> On Aug 21, 2015 12:02, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
>
>> I’ve noticed that this change has pretty much disabled Hadoop world
>> spefici precommit [1] hook as this job is running every 10 minutes and
>> fetching updates from JIRA, rather then following any repository. I’ve put
>> back the “run every 10 minutes”. This job is usually done in less then 30
>> seconds, so I’m assuming that it won’t be a problem :)
>>
>> Jarcec
>>
>> Links:
>> 1: https://builds.apache.org/job/PreCommit-Admin
>>
>>> On Aug 20, 2015, at 10:32 AM, Andrew Bayer <an...@gmail.com>
>> wrote:
>>>
>>> I noticed a lot of jobs that were running every day regardless of whether
>>> anything changed and eating up a lot of executors, so I bulk-removed all
>> of
>>> them, changing those jobs to poll for changes hourly instead. If your
>>> project has one or two jobs that you need to run daily whether there are
>>> code changes or not, you can re-enable the timer, but please do not do
>> that
>>> for more than a couple jobs, and please do not do it for jobs that take
>>> longer than half an hour.
>>>
>>> A.
>>
>>
Re: FYI - disabled timer triggers across the board on builds.apache.org
Posted by Andrew Bayer <an...@gmail.com>.
Oops, sorry about that!
On Aug 21, 2015 12:02, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
> I’ve noticed that this change has pretty much disabled Hadoop world
> spefici precommit [1] hook as this job is running every 10 minutes and
> fetching updates from JIRA, rather then following any repository. I’ve put
> back the “run every 10 minutes”. This job is usually done in less then 30
> seconds, so I’m assuming that it won’t be a problem :)
>
> Jarcec
>
> Links:
> 1: https://builds.apache.org/job/PreCommit-Admin
>
> > On Aug 20, 2015, at 10:32 AM, Andrew Bayer <an...@gmail.com>
> wrote:
> >
> > I noticed a lot of jobs that were running every day regardless of whether
> > anything changed and eating up a lot of executors, so I bulk-removed all
> of
> > them, changing those jobs to poll for changes hourly instead. If your
> > project has one or two jobs that you need to run daily whether there are
> > code changes or not, you can re-enable the timer, but please do not do
> that
> > for more than a couple jobs, and please do not do it for jobs that take
> > longer than half an hour.
> >
> > A.
>
>
Re: FYI - disabled timer triggers across the board on builds.apache.org
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
I’ve noticed that this change has pretty much disabled Hadoop world spefici precommit [1] hook as this job is running every 10 minutes and fetching updates from JIRA, rather then following any repository. I’ve put back the “run every 10 minutes”. This job is usually done in less then 30 seconds, so I’m assuming that it won’t be a problem :)
Jarcec
Links:
1: https://builds.apache.org/job/PreCommit-Admin
> On Aug 20, 2015, at 10:32 AM, Andrew Bayer <an...@gmail.com> wrote:
>
> I noticed a lot of jobs that were running every day regardless of whether
> anything changed and eating up a lot of executors, so I bulk-removed all of
> them, changing those jobs to poll for changes hourly instead. If your
> project has one or two jobs that you need to run daily whether there are
> code changes or not, you can re-enable the timer, but please do not do that
> for more than a couple jobs, and please do not do it for jobs that take
> longer than half an hour.
>
> A.
RE: FYI - disabled timer triggers across the board on builds.apache.org
Posted by Andrew Bayer <an...@gmail.com>.
Yeah, that's fine - the only viable way to do this was across the board, so
I'm sorry for messing with your jobs. =)
On Aug 20, 2015 17:41, "Uwe Schindler" <us...@apache.org> wrote:
> Hi Andrew,
>
> see my mail thread about the Lucene builds this morning with Gav:
>
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Thursday, August 20, 2015 11:28 AM
> > To: builds@apache.org
> > Subject: Re: Number of build prozessors on Lucene node
> >
> > Hi,
> >
> > The backlog of Lucene is generally large. That's wanted, because we don't
> > trigger builds based on commits; just to always have one build of
> everything
> > in the queue we do it time-based. Because of our pseudorandomized test
> > infrastructure, the build slave should never be out of work because every
> > run may trigger a bug in Lucene or the Java VM. Because Jenkins never
> > triggers the same job multiple times, it's perfectly fine to have at
> least one
> > job of all in the queue.
> >
> > If you don't want your statistics broken, you may exclude the lucene
> slave
> > from counting towards queue size. It's completely on its own and only
> allows
> > ecplicitely assigned jobs.
> >
> > You can always remove triggered builds from queu (I sometimes do this for
> > testing puposes or to trigger a special build manually). They get queued
> > anyways a while later if deleted.
> >
> > Uwe
> >
> > Am 20. August 2015 11:15:17 MESZ, schrieb Gavin McDonald
> > <gm...@apache.org>:
> > >
> > >> On 20 Aug 2015, at 9:17 am, Uwe Schindler <us...@apache.org>
> > >wrote:
> > >>
> > >> Hi,
> > >>
> > >> could someone please change back the number of "build processors" for
> > >the "lucene" Jenkins node to 1? Currently it executes always 2 Jobs in
> > >parallel. The underlying server only has 4 cpu cores and the Lucene Job
> > >configuration is done in a way that it uses all available CPU nodes, so
> > >running 2 builds in parallel on is in most cases not a good idea. There
> > >are in fact some jobs, that don't require much CPU or are not
> > >multithreaded (like artifact builds), but those are generally quick.
> > >The main tasks taking several hours to execute are very CPU intensive -
> > >the reason why we have an own slave.
> > >>
> > >> Any information why this was changed? Unfortunately I don't have the
> > >power to configure this correctly.
> > >
> > >I changed it short term to help with backlog.
> > >When I did this (2 days ago) there were 166 builds in the queue, over
> > >30 of those were waiting on the Lucene node.
> > >
> > >Will change it back now.
> > >
> > >HTH
> > >
> > >Gav…
> > >
> > >>
> > >> Uwe
> > >>
>
> As said before, the Lucene builds are only affecting the Lucene slave and
> this one should be used as much as possible, I reverted to the timer
> triggers. I am glad that you only changed builds which used timers,
> otherwise the whole cross-project dependency tree in Lucene would have been
> broken. So it was only a few jobs to start timely (per commit did not make
> sense for "nightly" jobs).
>
> Uwe
>
> -----
> Uwe Schindler
> uschindler@apache.org
> ASF Member, Apache Lucene PMC / Committer
> Bremen, Germany
> http://lucene.apache.org/
>
> > -----Original Message-----
> > From: Andrew Bayer [mailto:andrew.bayer@gmail.com]
> > Sent: Thursday, August 20, 2015 7:33 PM
> > To: builds@apache.org
> > Subject: FYI - disabled timer triggers across the board on
> builds.apache.org
> >
> > I noticed a lot of jobs that were running every day regardless of whether
> > anything changed and eating up a lot of executors, so I bulk-removed all
> of
> > them, changing those jobs to poll for changes hourly instead. If your
> project
> > has one or two jobs that you need to run daily whether there are code
> > changes or not, you can re-enable the timer, but please do not do that
> for
> > more than a couple jobs, and please do not do it for jobs that take
> longer
> > than half an hour.
> >
> > A.
>
>
RE: FYI - disabled timer triggers across the board on builds.apache.org
Posted by Uwe Schindler <us...@apache.org>.
Hi Andrew,
see my mail thread about the Lucene builds this morning with Gav:
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Thursday, August 20, 2015 11:28 AM
> To: builds@apache.org
> Subject: Re: Number of build prozessors on Lucene node
>
> Hi,
>
> The backlog of Lucene is generally large. That's wanted, because we don't
> trigger builds based on commits; just to always have one build of everything
> in the queue we do it time-based. Because of our pseudorandomized test
> infrastructure, the build slave should never be out of work because every
> run may trigger a bug in Lucene or the Java VM. Because Jenkins never
> triggers the same job multiple times, it's perfectly fine to have at least one
> job of all in the queue.
>
> If you don't want your statistics broken, you may exclude the lucene slave
> from counting towards queue size. It's completely on its own and only allows
> ecplicitely assigned jobs.
>
> You can always remove triggered builds from queu (I sometimes do this for
> testing puposes or to trigger a special build manually). They get queued
> anyways a while later if deleted.
>
> Uwe
>
> Am 20. August 2015 11:15:17 MESZ, schrieb Gavin McDonald
> <gm...@apache.org>:
> >
> >> On 20 Aug 2015, at 9:17 am, Uwe Schindler <us...@apache.org>
> >wrote:
> >>
> >> Hi,
> >>
> >> could someone please change back the number of "build processors" for
> >the "lucene" Jenkins node to 1? Currently it executes always 2 Jobs in
> >parallel. The underlying server only has 4 cpu cores and the Lucene Job
> >configuration is done in a way that it uses all available CPU nodes, so
> >running 2 builds in parallel on is in most cases not a good idea. There
> >are in fact some jobs, that don't require much CPU or are not
> >multithreaded (like artifact builds), but those are generally quick.
> >The main tasks taking several hours to execute are very CPU intensive -
> >the reason why we have an own slave.
> >>
> >> Any information why this was changed? Unfortunately I don't have the
> >power to configure this correctly.
> >
> >I changed it short term to help with backlog.
> >When I did this (2 days ago) there were 166 builds in the queue, over
> >30 of those were waiting on the Lucene node.
> >
> >Will change it back now.
> >
> >HTH
> >
> >Gav…
> >
> >>
> >> Uwe
> >>
As said before, the Lucene builds are only affecting the Lucene slave and this one should be used as much as possible, I reverted to the timer triggers. I am glad that you only changed builds which used timers, otherwise the whole cross-project dependency tree in Lucene would have been broken. So it was only a few jobs to start timely (per commit did not make sense for "nightly" jobs).
Uwe
-----
Uwe Schindler
uschindler@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/
> -----Original Message-----
> From: Andrew Bayer [mailto:andrew.bayer@gmail.com]
> Sent: Thursday, August 20, 2015 7:33 PM
> To: builds@apache.org
> Subject: FYI - disabled timer triggers across the board on builds.apache.org
>
> I noticed a lot of jobs that were running every day regardless of whether
> anything changed and eating up a lot of executors, so I bulk-removed all of
> them, changing those jobs to poll for changes hourly instead. If your project
> has one or two jobs that you need to run daily whether there are code
> changes or not, you can re-enable the timer, but please do not do that for
> more than a couple jobs, and please do not do it for jobs that take longer
> than half an hour.
>
> A.
Re: FYI - disabled timer triggers across the board on builds.apache.org
Posted by Andrew Bayer <an...@gmail.com>.
Hmm, I'll take a look - I'm restarting builds.a.o right now to deal with
the out of space issue.
A.
On Fri, Aug 21, 2015 at 12:31 PM, Stefan Seelmann <ma...@stefan-seelmann.de>
wrote:
> Polling would be fine for us, even @daily.
>
> But it seems the polling doesn't work, see [1]. There is only the icon
> but the "Subversion Polling Log" label is missing. When clicking on the
> icon it shown an error.
>
> For another job [2] I went into "Configure" and just saved without any
> further modification, then the polling seems to work.
>
> Kind Regards,
> Stefan
>
> [1] https://builds.apache.org/view/A-D/view/Directory/job/dir-studio/
>
> [2]
>
> https://builds.apache.org/view/A-D/view/Directory/job/dir-shared-ubuntu-deploy/
>
> On 08/20/2015 07:32 PM, Andrew Bayer wrote:
> > I noticed a lot of jobs that were running every day regardless of whether
> > anything changed and eating up a lot of executors, so I bulk-removed all
> of
> > them, changing those jobs to poll for changes hourly instead. If your
> > project has one or two jobs that you need to run daily whether there are
> > code changes or not, you can re-enable the timer, but please do not do
> that
> > for more than a couple jobs, and please do not do it for jobs that take
> > longer than half an hour.
> >
> > A.
> >
>
>
>
Re: FYI - disabled timer triggers across the board on
builds.apache.org
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 08/21/2015 06:50 PM, Andrew Bayer wrote:
> Ok, I think the restart seems to have cleared up the polling on jobs -
> sorry, I'll figure out a better way to handle this sort of thing in the
> future.
Thanks Andrew.
I changed all Directory jobs to poll @daily only, should be sufficient
for our needs.
Re: FYI - disabled timer triggers across the board on builds.apache.org
Posted by Andrew Bayer <an...@gmail.com>.
Ok, I think the restart seems to have cleared up the polling on jobs -
sorry, I'll figure out a better way to handle this sort of thing in the
future.
A.
On Fri, Aug 21, 2015 at 12:31 PM, Stefan Seelmann <ma...@stefan-seelmann.de>
wrote:
> Polling would be fine for us, even @daily.
>
> But it seems the polling doesn't work, see [1]. There is only the icon
> but the "Subversion Polling Log" label is missing. When clicking on the
> icon it shown an error.
>
> For another job [2] I went into "Configure" and just saved without any
> further modification, then the polling seems to work.
>
> Kind Regards,
> Stefan
>
> [1] https://builds.apache.org/view/A-D/view/Directory/job/dir-studio/
>
> [2]
>
> https://builds.apache.org/view/A-D/view/Directory/job/dir-shared-ubuntu-deploy/
>
> On 08/20/2015 07:32 PM, Andrew Bayer wrote:
> > I noticed a lot of jobs that were running every day regardless of whether
> > anything changed and eating up a lot of executors, so I bulk-removed all
> of
> > them, changing those jobs to poll for changes hourly instead. If your
> > project has one or two jobs that you need to run daily whether there are
> > code changes or not, you can re-enable the timer, but please do not do
> that
> > for more than a couple jobs, and please do not do it for jobs that take
> > longer than half an hour.
> >
> > A.
> >
>
>
>
Re: FYI - disabled timer triggers across the board on
builds.apache.org
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
Polling would be fine for us, even @daily.
But it seems the polling doesn't work, see [1]. There is only the icon
but the "Subversion Polling Log" label is missing. When clicking on the
icon it shown an error.
For another job [2] I went into "Configure" and just saved without any
further modification, then the polling seems to work.
Kind Regards,
Stefan
[1] https://builds.apache.org/view/A-D/view/Directory/job/dir-studio/
[2]
https://builds.apache.org/view/A-D/view/Directory/job/dir-shared-ubuntu-deploy/
On 08/20/2015 07:32 PM, Andrew Bayer wrote:
> I noticed a lot of jobs that were running every day regardless of whether
> anything changed and eating up a lot of executors, so I bulk-removed all of
> them, changing those jobs to poll for changes hourly instead. If your
> project has one or two jobs that you need to run daily whether there are
> code changes or not, you can re-enable the timer, but please do not do that
> for more than a couple jobs, and please do not do it for jobs that take
> longer than half an hour.
>
> A.
>