You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Gavin McDonald <ga...@16degrees.com.au> on 2016/11/10 09:22:25 UTC
[IMPORTANT] - A lot of jenkins jobs have been changed!
Hi All,
Yesterday whilst adding some new modes and re-configuring jobs & nodes for removal of some Labels - as notified some weeks ago, I noticed some
unusual configurations that could affect the overall performance of Jenkins and affect everybody in general.
The changes were wide spread and out of over 2000 builds I would say at least 1500 of them are affected in one way or another.
To point out, none of this is drastic, and your builds are not affected in terms of software configurations. But a hammer was used here to
change a lot of builds at once, to make them more uniform.
A few things were changed.
1. Quite a few builds were reduced to keep a maximum of 5 last builds. This is mainly a space saving effort. (see note below)
- this may sound not much but some projects had builds configured to keep 500, 300, 200, 100, 75, 50, 30 builds etc. so it
really adds up. I understand that some projects may want to keep builds around longer
2. Some builds were also changed to keep a maximum of 14 days worth of past builds. Again a space saving effort. As an example some
projects were configured to keep 2000, 1000, 800, 500, 200 jobs etc etc ..
3. Timeout minutes. A fair few builds involved here, some jobs exceeding what *I thought* was sensible were pruned to a maximum of one
or two hours here. Some jobs for example were configured for 800, 600, etc minutes. One job was set to timeout after 12000 minutes.
I understand that some jobs may break as a result of my changes and obviously that is not the outcome we are aiming for so please
do feel free to change your job back to a more sensible setting for your project (And optionally send a mail to builds@ saying you have
done so and why).
4. The same things were done with keep builds with artefacts, reduced to sensible numbers where there were some ‘out there’ configs.
Due to some of these reductions, over 400GB has been saved so far in terms of disk. Stuck jobs should also abort at more sensible
times too.
I’ll talk about the labels and new nodes in a different email.
Any questions, comments, concerns etc please do feel free to email back to the builds@apache.org <ma...@apache.org> list.
I apologise for the hammer affect, I did not have time to check stats for 2000+ jobs.
I also apologise for the lack of notice and only notifying after the fact but it needed doing.
Thanks
Gav… (ASF Infra)
Re: [IMPORTANT] - A lot of jenkins jobs have been changed!
Posted by Bertrand Delacretaz <bd...@apache.org>.
On Mon, Nov 14, 2016 at 12:59 PM, Robert Munteanu <ro...@apache.org> wrote:
> ...there's also a DSL -> config.xml converter online at [3], so you
> can see the results of your DSL before adding it to SVN...
And Docker images like this one are useful for testing Jenkins DSL and
other scripts on your own boxes:
https://github.com/thomasleveil/docker-jenkins-dsl-ready
-Bertrand
Re: [IMPORTANT] - A lot of jenkins jobs have been changed!
Posted by Robert Munteanu <ro...@apache.org>.
On Fri, 2016-11-11 at 04:52 +0100, Herv� BOUTEMY wrote:
(snip again)
> > 2) I would encourage more projects to manage their Jenkins
> > configurations in version control, either by using a Jenkinsfile or
> > the
> > Job DSL plugin, it makes it much simpler to maintain your
> > configuration
> > in time and easier to collaborate on.
>
> this is something that I'm interested to do with Maven, but I'd need:
>
> - some pointers on how to do it: I'm not a Jenkins expert, and I
> suppose I'm�
> not the only one, then this could be useful for others too
> found [1]: that looks a good start: thank you Robert :)
> I don't know if this should be generalized, nor where to put
> information�
> (MoinMoin Jenkins Wiki page, or new infra pages on Jenkins nodes...)
The Job DSL plug-in documentation is pretty good, you can start with
https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin
The main thing to wrap your head around is that there will be a 'seed'
job in Jenkins which manages the rest of the jobs. This is usually a
freestyle job which invokes a 'Process DSL' build step. We keep the
config in SVN, which is an added bonus.
Other than that, the API docs are pretty good, you can see them at [2]
and there's also a DSL -> config.xml converter online at [3], so you
can see the results of your DSL before adding it to SVN.
Hope this helps,
Robert
> > [0]: https://builds.apache.org/view/S-Z/view/Sling/
>
> [1] https://cwiki.apache.org/confluence/display/SLING/Sling+Jenkins+S
> etup
[2]: https://jenkinsci.github.io/job-dsl-plugin/#
[3]: http://job-dsl.herokuapp.com/
Re: [IMPORTANT] - A lot of jenkins jobs have been changed!
Posted by Hervé BOUTEMY <he...@free.fr>.
Le jeudi 10 novembre 2016, 15:40:03 CET Robert Munteanu a écrit :
(snip)
>
> Thanks for taking care of the overall health of the Jenkins system,
> much appreciated.
+1
> In the Apache Sling project we're managing a sizeable number of jobs (
> 530 at the moment ) [0] using the Job DSL Plugin to make it feasible to
> look after such a large number of jobs.
>
> The jobs are quite small, usually don't archive artifacts ( we deploy
> to repository.apache.org ) and many of them are triggered rarely, so
> they should not have a negative impact on the health of the Jenkins
> server. I rather suspect we have reduced our load since we rebuild a
> much smaller part of the project whenever anything changes.
>
> I want to point out 2 things:
>
> 1) If you think that we're doing something wrong in the project configs
> do let us know separately - the manual changes you might apply will be
> overwritten without us noticing.
>
> 2) I would encourage more projects to manage their Jenkins
> configurations in version control, either by using a Jenkinsfile or the
> Job DSL plugin, it makes it much simpler to maintain your configuration
> in time and easier to collaborate on.
this is something that I'm interested to do with Maven, but I'd need:
- some pointers on how to do it: I'm not a Jenkins expert, and I suppose I'm
not the only one, then this could be useful for others too
found [1]: that looks a good start: thank you Robert :)
I don't know if this should be generalized, nor where to put information
(MoinMoin Jenkins Wiki page, or new infra pages on Jenkins nodes...)
- some feedback if this really won't affect the Jenkins instance performance
if every project does the same: the number of (little) jobs will grow a lot,
to replace one big job that rebuild in 1 row a lot of independant parts. I see
for example that Sling defined 2 views to separate "build view" (with many many
jobs) from "dashboard for builds that need attention" (with the few currently
"interesting" ones)
Any objection for me to start doing the same on Maven Jenkins jobs?
Regards,
Hervé
>
> Best,
>
> Robert
>
> [0]: https://builds.apache.org/view/S-Z/view/Sling/
[1] https://cwiki.apache.org/confluence/display/SLING/Sling+Jenkins+Setup
Re: [IMPORTANT] - A lot of jenkins jobs have been changed!
Posted by Gavin McDonald <ga...@16degrees.com.au>.
> On 11 Nov. 2016, at 10:10 am, Michael Shuler <mi...@pbandjelly.org> wrote:
>
> On 11/10/2016 04:29 PM, Gavin McDonald wrote:
>>> 2) I would encourage more projects to manage their Jenkins
>>> configurations in version control, either by using a Jenkinsfile or
>>> the Job DSL plugin, it makes it much simpler to maintain your
>>> configuration in time and easier to collaborate on.
>
> We're using DSL internally and I am planning on setting up the Cassandra
> jobs to DSL - works really well.
>
>> Thats a good idea. (In which case you’ll be able to compare the
>> changes made with your svn copy and adjust as needed?)
>
> Unfortunately not easily. Any config edit or `sed` updates of the
> config.xml will just get overwritten by DSL, unless someone is notified
> to go look in time to see what was changed and get it into the DSL VCS.
>
Thanks, handy to know.
> (The `sed` of 's/docker/ubunt/' sort of broke things.. There are some
> jobs in the queue now with nowhere to run.)
12 builds affected, thanks for pointing it out, now fixed.
As asn aside, I use a Jenkins plugin to make the edits, not ‘sed’. :)
>
> I had planned to update the banners on the jobs that they were using DSL
> with a link to the repository, to hopefully let people know that any
> hand edits are only temporary. I'm not really sure of a better way, but
> I'm up for ideas.
I dont have any better ideas currently.
My recent hammer action is a one off, we may check again in 6 months or so
but I can’t imagine we would need to do this again for a long time.
Gav…
>
> --
> Kind regards,
> Michael
>
Re: [IMPORTANT] - A lot of jenkins jobs have been changed!
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 11/10/2016 04:29 PM, Gavin McDonald wrote:
>> 2) I would encourage more projects to manage their Jenkins
>> configurations in version control, either by using a Jenkinsfile or
>> the Job DSL plugin, it makes it much simpler to maintain your
>> configuration in time and easier to collaborate on.
We're using DSL internally and I am planning on setting up the Cassandra
jobs to DSL - works really well.
> Thats a good idea. (In which case you\u2019ll be able to compare the
> changes made with your svn copy and adjust as needed?)
Unfortunately not easily. Any config edit or `sed` updates of the
config.xml will just get overwritten by DSL, unless someone is notified
to go look in time to see what was changed and get it into the DSL VCS.
(The `sed` of 's/docker/ubunt/' sort of broke things.. There are some
jobs in the queue now with nowhere to run.)
I had planned to update the banners on the jobs that they were using DSL
with a link to the repository, to hopefully let people know that any
hand edits are only temporary. I'm not really sure of a better way, but
I'm up for ideas.
--
Kind regards,
Michael
Re: [IMPORTANT] - A lot of jenkins jobs have been changed!
Posted by Gavin McDonald <ga...@16degrees.com.au>.
Hi Robert,
> On 11 Nov. 2016, at 1:40 am, Robert Munteanu <ro...@apache.org> wrote:
>
> Hi Gavin,
>
> On Thu, 2016-11-10 at 20:22 +1100, Gavin McDonald wrote:
>> Hi All,
>>
>> Yesterday whilst adding some new modes and re-configuring jobs &
>> nodes for removal of some Labels - as notified some weeks ago, I
>> noticed some
>> unusual configurations that could affect the overall performance of
>> Jenkins and affect everybody in general.
>>
>> The changes were wide spread and out of over 2000 builds I would say
>> at least 1500 of them are affected in one way or another.
>> To point out, none of this is drastic, and your builds are not
>> affected in terms of software configurations. But a hammer was used
>> here to
>> change a lot of builds at once, to make them more uniform.
>>
>> A few things were changed.
>
> (snip)
>
> Thanks for taking care of the overall health of the Jenkins system,
> much appreciated.
Thank You
>
> In the Apache Sling project we're managing a sizeable number of jobs (
> 530 at the moment ) [0] using the Job DSL Plugin to make it feasible to
> look after such a large number of jobs.
Yes I noticed the amount of Sling jobs :) I even went as far as to see
how you had them all programmed.
>
> The jobs are quite small, usually don't archive artifacts ( we deploy
> to repository.apache.org ) and many of them are triggered rarely, so
> they should not have a negative impact on the health of the Jenkins
> server. I rather suspect we have reduced our load since we rebuild a
> much smaller part of the project whenever anything changes.
Good to know.
>
> I want to point out 2 things:
>
> 1) If you think that we're doing something wrong in the project configs
> do let us know separately - the manual changes you might apply will be
> overwritten without us noticing.
>
Sure, this was a rare ‘act now, notify later’. And in this instance the changes
were so widespread, notifying each project seperately would have been a
big drain on time.
In fact, this list (builds@apache.org <ma...@apache.org>) is the main focussed list that all projects
need to have representation on. I have in the recent past sent our important
mails to individual projects - and at the same time let them know that future
mails that affected them would come to the builds list instead. It is much
easier to focus on one list for correspondence than many.
> 2) I would encourage more projects to manage their Jenkins
> configurations in version control, either by using a Jenkinsfile or the
> Job DSL plugin, it makes it much simpler to maintain your configuration
> in time and easier to collaborate on.
>
Thats a good idea. (In which case you’ll be able to compare the changes
made with your svn copy and adjust as needed?)
Gav…
> Best,
>
> Robert
>
> [0]: https://builds.apache.org/view/S-Z/view/Sling/
Re: [IMPORTANT] - A lot of jenkins jobs have been changed!
Posted by Robert Munteanu <ro...@apache.org>.
Hi Gavin,
On Thu, 2016-11-10 at 20:22 +1100, Gavin McDonald wrote:
> Hi All,
>
> Yesterday whilst adding some new modes and re-configuring jobs &
> nodes for removal of some Labels - as notified some weeks ago, I
> noticed some�
> unusual configurations that could affect the overall performance of
> Jenkins and affect everybody in general.
>
> The changes were wide spread and out of over 2000 builds I would say
> at least 1500 of them are affected in one way or another.
> To point out, none of this is drastic, and your builds are not
> affected in terms of software configurations. But a hammer was used
> here to�
> change a lot of builds at once, to make them more uniform.
>
> A few things were changed.
(snip)
Thanks for taking care of the overall health of the Jenkins system,
much appreciated.
In the Apache Sling project we're managing a sizeable number of jobs (
530 at the moment ) [0] using the Job DSL Plugin to make it feasible to
look after such a large number of jobs.
The jobs are quite small, usually don't archive artifacts ( we deploy
to repository.apache.org ) and many of them are triggered rarely, so
they should not have a negative impact on the health of the Jenkins
server. I rather suspect we have reduced our load since we rebuild a
much smaller part of the project whenever anything changes.
I want to point out 2 things:
1) If you think that we're doing something wrong in the project configs
do let us know separately - the manual changes you might apply will be
overwritten without us noticing.
2) I would encourage more projects to manage their Jenkins
configurations in version control, either by using a Jenkinsfile or the
Job DSL plugin, it makes it much simpler to maintain your configuration
in time and easier to collaborate on.
Best,
Robert
[0]: https://builds.apache.org/view/S-Z/view/Sling/
Re: [IMPORTANT] - A lot of jenkins jobs have been changed!
Posted by Lukasz Lenart <lu...@apache.org>.
This is osm! Thanks a lot!
2016-11-10 10:22 GMT+01:00 Gavin McDonald <ga...@16degrees.com.au>:
> Hi All,
>
> Yesterday whilst adding some new modes and re-configuring jobs & nodes for removal of some Labels - as notified some weeks ago, I noticed some
> unusual configurations that could affect the overall performance of Jenkins and affect everybody in general.
>
> The changes were wide spread and out of over 2000 builds I would say at least 1500 of them are affected in one way or another.
> To point out, none of this is drastic, and your builds are not affected in terms of software configurations. But a hammer was used here to
> change a lot of builds at once, to make them more uniform.
>
> A few things were changed.
>
> 1. Quite a few builds were reduced to keep a maximum of 5 last builds. This is mainly a space saving effort. (see note below)
> - this may sound not much but some projects had builds configured to keep 500, 300, 200, 100, 75, 50, 30 builds etc. so it
> really adds up. I understand that some projects may want to keep builds around longer
>
> 2. Some builds were also changed to keep a maximum of 14 days worth of past builds. Again a space saving effort. As an example some
> projects were configured to keep 2000, 1000, 800, 500, 200 jobs etc etc ..
>
> 3. Timeout minutes. A fair few builds involved here, some jobs exceeding what *I thought* was sensible were pruned to a maximum of one
> or two hours here. Some jobs for example were configured for 800, 600, etc minutes. One job was set to timeout after 12000 minutes.
> I understand that some jobs may break as a result of my changes and obviously that is not the outcome we are aiming for so please
> do feel free to change your job back to a more sensible setting for your project (And optionally send a mail to builds@ saying you have
> done so and why).
>
> 4. The same things were done with keep builds with artefacts, reduced to sensible numbers where there were some ‘out there’ configs.
>
> Due to some of these reductions, over 400GB has been saved so far in terms of disk. Stuck jobs should also abort at more sensible
> times too.
>
> I’ll talk about the labels and new nodes in a different email.
>
> Any questions, comments, concerns etc please do feel free to email back to the builds@apache.org <ma...@apache.org> list.
>
> I apologise for the hammer affect, I did not have time to check stats for 2000+ jobs.
> I also apologise for the lack of notice and only notifying after the fact but it needed doing.
>
> Thanks
>
> Gav… (ASF Infra)
>
>