You are viewing a plain text version of this content. The canonical link for it is here.

Posted to builds@apache.org by Dennis Kieselhorst <de...@apache.org> on 2020/08/03 12:13:01 UTC

Re: Maven builds out of control

Hi,

currently the queue is again ~3500 builds long. I already experienced this several times. Any update on a fix?

Best,
Dennis

Re: Maven builds out of control

Posted by sebb <se...@gmail.com>.

Still thousands of queues jobs waiting on Windows and Ubuntu nodes

Most of the nodes seem to be processing Maven jobs.

Some have been running for several hours now.

On Mon, 3 Aug 2020 at 13:21, Olivier Lamy <ol...@apache.org> wrote:
>
> I changed a configuration to reduce maven jobs disk saving size.
> no worries builds will be absorbed quickly.
>
> On Mon, 3 Aug 2020 at 14:13, Dennis Kieselhorst <de...@apache.org> wrote:
>
> > Hi,
> >
> > currently the queue is again ~3500 builds long. I already experienced this
> > several times. Any update on a fix?
> >
> > Best,
> > Dennis
> >
>
>
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy

Re: Fwd: Maven builds out of control

Posted by Arnaud Héritier <ah...@gmail.com>.

No I agree, it's a long term solution and orthogonal on cleaning our
branches/builds and limiting our usages

If these are static agents the best we could do at the CI level is to
restrict the usage of our builds to a limited number of agents
https://docs.cloudbees.com/docs/cloudbees-ci/latest/traditional-secure-guide/folders-plus#_controlled_agents


On Mon, Aug 17, 2020 at 10:55 AM Olivier Lamy <ol...@apache.org> wrote:

> traditional slaves
> I have a goal to help moving to kubernetes but this will not fix this
> problem :)
>
>
> On Mon, 17 Aug 2020 at 16:46, Arnaud Héritier <ah...@gmail.com> wrote:
>
> > Thanks for the feedback Gavin, it makes a lot of sense to me.
> > The new infrastructure is a traditional one (ie not hosted on
> Kubernetes) ?
> >
> > On Mon, Aug 17, 2020 at 10:44 AM Gavin McDonald <gm...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <ol...@apache.org>
> wrote:
> > >
> > > > +infra
> > > >
> > > > On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <aheritier@gmail.com
> >
> > > > wrote:
> > > >
> > > >> If we are consuming too much resources couldn't we have a dedicated
> > > >> Jenkins
> > > >>
> > > >> controller and/or limit our agents capacities to be the one ones to
> be
> > > >>
> > > >> impacted by these problems instead of all the users or this Jenkins
> > > >>
> > > >> controller ?
> > > >>
> > > >> Beam, Cassandra, CouchDB and few more have a dedicated controller :
> > > >>
> > > >> https://jenkins-ccos.apache.org/job/masters/
> > > >>
> > > >> What are the rules to have a dedicated one ?
> > > >>
> > > >
> > > All of those projects got their own master as they are contributing
> their
> > > own nodes and not using the ASF provided ones
> > >
> > > HTH
> > >
> > >
> > > >
> > > >>
> > > >>
> > > >> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <
> michaelo@apache.org>
> > > >> wrote:
> > > >>
> > > >>
> > > >>
> > > >> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> > > >>
> > > >> > > Furthermore, we need to clean up unused branches.
> > > >>
> > > >> > > What about removing branches older than 1yo?
> > > >>
> > > >> > > Then older than 6months?
> > > >>
> > > >> >
> > > >>
> > > >> > I consider branches which have been merged obsolete, but there is
> no
> > > >>
> > > >> > hard rule that an unmerged branch is obsolete. It may prepresent
> an
> > > >>
> > > >> > unfished state or something which was not agreed on. Dropping work
> > > does
> > > >>
> > > >> > not feel right. It needs to be decided on a case-by-case approach.
> > > >>
> > > >> > I have done this recently for Wagon and Resolver.
> > > >>
> > > >> >
> > > >>
> > > >> > Michael
> > > >>
> > > >> >
> > > >>
> > > >> >
> > ---------------------------------------------------------------------
> > > >>
> > > >> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >>
> > > >> > For additional commands, e-mail: dev-help@maven.apache.org
> > > >>
> > > >> >
> > > >>
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Arnaud Héritier
> > > >>
> > > >> Twitter/Skype : aheritier
> > > >>
> > > >> --
> > > > Olivier Lamy
> > > > http://twitter.com/olamy | http://linkedin.com/in/olamy
> > > >
> > >
> > >
> > > --
> > >
> > > *Gavin McDonald*
> > > Systems Administrator
> > > ASF Infrastructure Team
> > >
> >
> >
> > --
> > Arnaud Héritier
> > Twitter/Skype : aheritier
> >
>
>
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>


-- 
Arnaud Héritier
Twitter/Skype : aheritier

Re: Fwd: Maven builds out of control

Posted by Olivier Lamy <ol...@apache.org>.

traditional slaves
I have a goal to help moving to kubernetes but this will not fix this
problem :)


On Mon, 17 Aug 2020 at 16:46, Arnaud Héritier <ah...@gmail.com> wrote:

> Thanks for the feedback Gavin, it makes a lot of sense to me.
> The new infrastructure is a traditional one (ie not hosted on Kubernetes) ?
>
> On Mon, Aug 17, 2020 at 10:44 AM Gavin McDonald <gm...@apache.org>
> wrote:
>
> > Hi,
> >
> > On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <ol...@apache.org> wrote:
> >
> > > +infra
> > >
> > > On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <ah...@gmail.com>
> > > wrote:
> > >
> > >> If we are consuming too much resources couldn't we have a dedicated
> > >> Jenkins
> > >>
> > >> controller and/or limit our agents capacities to be the one ones to be
> > >>
> > >> impacted by these problems instead of all the users or this Jenkins
> > >>
> > >> controller ?
> > >>
> > >> Beam, Cassandra, CouchDB and few more have a dedicated controller :
> > >>
> > >> https://jenkins-ccos.apache.org/job/masters/
> > >>
> > >> What are the rules to have a dedicated one ?
> > >>
> > >
> > All of those projects got their own master as they are contributing their
> > own nodes and not using the ASF provided ones
> >
> > HTH
> >
> >
> > >
> > >>
> > >>
> > >> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org>
> > >> wrote:
> > >>
> > >>
> > >>
> > >> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> > >>
> > >> > > Furthermore, we need to clean up unused branches.
> > >>
> > >> > > What about removing branches older than 1yo?
> > >>
> > >> > > Then older than 6months?
> > >>
> > >> >
> > >>
> > >> > I consider branches which have been merged obsolete, but there is no
> > >>
> > >> > hard rule that an unmerged branch is obsolete. It may prepresent an
> > >>
> > >> > unfished state or something which was not agreed on. Dropping work
> > does
> > >>
> > >> > not feel right. It needs to be decided on a case-by-case approach.
> > >>
> > >> > I have done this recently for Wagon and Resolver.
> > >>
> > >> >
> > >>
> > >> > Michael
> > >>
> > >> >
> > >>
> > >> >
> ---------------------------------------------------------------------
> > >>
> > >> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > >>
> > >> > For additional commands, e-mail: dev-help@maven.apache.org
> > >>
> > >> >
> > >>
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Arnaud Héritier
> > >>
> > >> Twitter/Skype : aheritier
> > >>
> > >> --
> > > Olivier Lamy
> > > http://twitter.com/olamy | http://linkedin.com/in/olamy
> > >
> >
> >
> > --
> >
> > *Gavin McDonald*
> > Systems Administrator
> > ASF Infrastructure Team
> >
>
>
> --
> Arnaud Héritier
> Twitter/Skype : aheritier
>


-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

Re: Fwd: Maven builds out of control

Posted by Arnaud Héritier <ah...@gmail.com>.

Thanks for the feedback Gavin, it makes a lot of sense to me.
The new infrastructure is a traditional one (ie not hosted on Kubernetes) ?

On Mon, Aug 17, 2020 at 10:44 AM Gavin McDonald <gm...@apache.org>
wrote:

> Hi,
>
> On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <ol...@apache.org> wrote:
>
> > +infra
> >
> > On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <ah...@gmail.com>
> > wrote:
> >
> >> If we are consuming too much resources couldn't we have a dedicated
> >> Jenkins
> >>
> >> controller and/or limit our agents capacities to be the one ones to be
> >>
> >> impacted by these problems instead of all the users or this Jenkins
> >>
> >> controller ?
> >>
> >> Beam, Cassandra, CouchDB and few more have a dedicated controller :
> >>
> >> https://jenkins-ccos.apache.org/job/masters/
> >>
> >> What are the rules to have a dedicated one ?
> >>
> >
> All of those projects got their own master as they are contributing their
> own nodes and not using the ASF provided ones
>
> HTH
>
>
> >
> >>
> >>
> >> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org>
> >> wrote:
> >>
> >>
> >>
> >> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> >>
> >> > > Furthermore, we need to clean up unused branches.
> >>
> >> > > What about removing branches older than 1yo?
> >>
> >> > > Then older than 6months?
> >>
> >> >
> >>
> >> > I consider branches which have been merged obsolete, but there is no
> >>
> >> > hard rule that an unmerged branch is obsolete. It may prepresent an
> >>
> >> > unfished state or something which was not agreed on. Dropping work
> does
> >>
> >> > not feel right. It needs to be decided on a case-by-case approach.
> >>
> >> > I have done this recently for Wagon and Resolver.
> >>
> >> >
> >>
> >> > Michael
> >>
> >> >
> >>
> >> > ---------------------------------------------------------------------
> >>
> >> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>
> >> > For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >> >
> >>
> >> >
> >>
> >>
> >>
> >> --
> >>
> >> Arnaud Héritier
> >>
> >> Twitter/Skype : aheritier
> >>
> >> --
> > Olivier Lamy
> > http://twitter.com/olamy | http://linkedin.com/in/olamy
> >
>
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
>


-- 
Arnaud Héritier
Twitter/Skype : aheritier

Re: Fwd: Maven builds out of control

Posted by Gavin McDonald <gm...@apache.org>.

Hi,

On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <ol...@apache.org> wrote:

> +infra
>
> On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <ah...@gmail.com>
> wrote:
>
>> If we are consuming too much resources couldn't we have a dedicated
>> Jenkins
>>
>> controller and/or limit our agents capacities to be the one ones to be
>>
>> impacted by these problems instead of all the users or this Jenkins
>>
>> controller ?
>>
>> Beam, Cassandra, CouchDB and few more have a dedicated controller :
>>
>> https://jenkins-ccos.apache.org/job/masters/
>>
>> What are the rules to have a dedicated one ?
>>
>
All of those projects got their own master as they are contributing their
own nodes and not using the ASF provided ones

HTH


>
>>
>>
>> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org>
>> wrote:
>>
>>
>>
>> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
>>
>> > > Furthermore, we need to clean up unused branches.
>>
>> > > What about removing branches older than 1yo?
>>
>> > > Then older than 6months?
>>
>> >
>>
>> > I consider branches which have been merged obsolete, but there is no
>>
>> > hard rule that an unmerged branch is obsolete. It may prepresent an
>>
>> > unfished state or something which was not agreed on. Dropping work does
>>
>> > not feel right. It needs to be decided on a case-by-case approach.
>>
>> > I have done this recently for Wagon and Resolver.
>>
>> >
>>
>> > Michael
>>
>> >
>>
>> > ---------------------------------------------------------------------
>>
>> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>
>> > For additional commands, e-mail: dev-help@maven.apache.org
>>
>> >
>>
>> >
>>
>>
>>
>> --
>>
>> Arnaud Héritier
>>
>> Twitter/Skype : aheritier
>>
>> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>


-- 

*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team

Re: Fwd: Maven builds out of control

Posted by Olivier Lamy <ol...@apache.org>.

+infra

On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <ah...@gmail.com> wrote:

> If we are consuming too much resources couldn't we have a dedicated Jenkins
>
> controller and/or limit our agents capacities to be the one ones to be
>
> impacted by these problems instead of all the users or this Jenkins
>
> controller ?
>
> Beam, Cassandra, CouchDB and few more have a dedicated controller :
>
> https://jenkins-ccos.apache.org/job/masters/
>
> What are the rules to have a dedicated one ?
>
>
>
> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org>
> wrote:
>
>
>
> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
>
> > > Furthermore, we need to clean up unused branches.
>
> > > What about removing branches older than 1yo?
>
> > > Then older than 6months?
>
> >
>
> > I consider branches which have been merged obsolete, but there is no
>
> > hard rule that an unmerged branch is obsolete. It may prepresent an
>
> > unfished state or something which was not agreed on. Dropping work does
>
> > not feel right. It needs to be decided on a case-by-case approach.
>
> > I have done this recently for Wagon and Resolver.
>
> >
>
> > Michael
>
> >
>
> > ---------------------------------------------------------------------
>
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>
> > For additional commands, e-mail: dev-help@maven.apache.org
>
> >
>
> >
>
>
>
> --
>
> Arnaud Héritier
>
> Twitter/Skype : aheritier
>
> --
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

Re: Fwd: Maven builds out of control

Posted by Arnaud Héritier <ah...@gmail.com>.

If we are consuming too much resources couldn't we have a dedicated Jenkins
controller and/or limit our agents capacities to be the one ones to be
impacted by these problems instead of all the users or this Jenkins
controller ?
Beam, Cassandra, CouchDB and few more have a dedicated controller :
https://jenkins-ccos.apache.org/job/masters/
What are the rules to have a dedicated one ?

On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org> wrote:

> Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> > Furthermore, we need to clean up unused branches.
> > What about removing branches older than 1yo?
> > Then older than 6months?
>
> I consider branches which have been merged obsolete, but there is no
> hard rule that an unmerged branch is obsolete. It may prepresent an
> unfished state or something which was not agreed on. Dropping work does
> not feel right. It needs to be decided on a case-by-case approach.
> I have done this recently for Wagon and Resolver.
>
> Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>

-- 
Arnaud Héritier
Twitter/Skype : aheritier

Re: Fwd: Maven builds out of control

Posted by Michael Osipov <mi...@apache.org>.

Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> Furthermore, we need to clean up unused branches.
> What about removing branches older than 1yo?
> Then older than 6months?

I consider branches which have been merged obsolete, but there is no 
hard rule that an unmerged branch is obsolete. It may prepresent an 
unfished state or something which was not agreed on. Dropping work does 
not feel right. It needs to be decided on a case-by-case approach.
I have done this recently for Wagon and Resolver.

Michael

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Re: Fwd: Maven builds out of control

Posted by Olivier Lamy <ol...@apache.org>.

Yes, we can limit branches using regex.
if use @dependabot merge the branch is deleted


On Mon, 17 Aug 2020 at 16:41, Maarten Mulders <mt...@apache.org> wrote:

> On 17/08/2020 09:55, Olivier Lamy wrote:
> > Hi there
> > we definitely need to reduce the matrix we have and cleanup branches ...
> > so I reduced the matrix (no more 14,15)
> > and only '3.3.x','3.6.x'
> >
> > Furthermore, we need to clean up unused branches.
> > What about removing branches older than 1yo?
> > Then older than 6months?
>
> Nowadays there are a lot of branches created by Dependabot.
> I think it's good to update dependencies more frequently than we
> currently do. But we shouldn't be building all those branches on Jenkins
> immediately, and without (human) review. There's a risk of unreviewed
> code being ran on Jenkins. Also, as we can see from this situation, it
> creates so many branches that are never cleaned up.
>
> If there would be a way to prevent those branches being built, that
> would definitely save space in the build queue. All branches start with
> dependabot/, but I haven't found a way yet to filter those out.
>
> Thanks,
>
> Maarten
>
> > ---------- Forwarded message ---------
> > From: Richard Eckart de Castilho <re...@apache.org>
> > Date: Mon, 17 Aug 2020 at 15:37
> > Subject: Re: Maven builds out of control
> > To: <bu...@apache.org>
> >
> >
> > Dear Maven developers,
> >
> > please fix or disable your CI builds ... other people need to use the
> > servers as well and in particular those who need to test jobs as part of
> > their job migration.
> >
> > @INFRA: Maybe confine all Maven Jobs to a single specific slave for the
> > time being? That way, they only starve themselves.
> >
> > Cheers,
> >
> > -- Richard
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>

-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy

Re: Fwd: Maven builds out of control

Posted by Maarten Mulders <mt...@apache.org>.

On 17/08/2020 09:55, Olivier Lamy wrote:
> Hi there
> we definitely need to reduce the matrix we have and cleanup branches ...
> so I reduced the matrix (no more 14,15)
> and only '3.3.x','3.6.x'
> 
> Furthermore, we need to clean up unused branches.
> What about removing branches older than 1yo?
> Then older than 6months?

Nowadays there are a lot of branches created by Dependabot.
I think it's good to update dependencies more frequently than we 
currently do. But we shouldn't be building all those branches on Jenkins 
immediately, and without (human) review. There's a risk of unreviewed 
code being ran on Jenkins. Also, as we can see from this situation, it 
creates so many branches that are never cleaned up.

If there would be a way to prevent those branches being built, that 
would definitely save space in the build queue. All branches start with 
dependabot/, but I haven't found a way yet to filter those out.

Thanks,

Maarten

> ---------- Forwarded message ---------
> From: Richard Eckart de Castilho <re...@apache.org>
> Date: Mon, 17 Aug 2020 at 15:37
> Subject: Re: Maven builds out of control
> To: <bu...@apache.org>
> 
> 
> Dear Maven developers,
> 
> please fix or disable your CI builds ... other people need to use the
> servers as well and in particular those who need to test jobs as part of
> their job migration.
> 
> @INFRA: Maybe confine all Maven Jobs to a single specific slave for the
> time being? That way, they only starve themselves.
> 
> Cheers,
> 
> -- Richard
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org

Fwd: Maven builds out of control

Posted by Olivier Lamy <ol...@apache.org>.

Hi there
we definitely need to reduce the matrix we have and cleanup branches ...
so I reduced the matrix (no more 14,15)
and only '3.3.x','3.6.x'

Furthermore, we need to clean up unused branches.
What about removing branches older than 1yo?
Then older than 6months?


---------- Forwarded message ---------
From: Richard Eckart de Castilho <re...@apache.org>
Date: Mon, 17 Aug 2020 at 15:37
Subject: Re: Maven builds out of control
To: <bu...@apache.org>


Dear Maven developers,

please fix or disable your CI builds ... other people need to use the
servers as well and in particular those who need to test jobs as part of
their job migration.

@INFRA: Maybe confine all Maven Jobs to a single specific slave for the
time being? That way, they only starve themselves.

Cheers,

-- Richard


-- 
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy