You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@maven.apache.org by Olivier Lamy <ol...@apache.org> on 2020/08/17 07:55:46 UTC
Fwd: Maven builds out of control
Hi there
we definitely need to reduce the matrix we have and cleanup branches ...
so I reduced the matrix (no more 14,15)
and only '3.3.x','3.6.x'
Furthermore, we need to clean up unused branches.
What about removing branches older than 1yo?
Then older than 6months?
---------- Forwarded message ---------
From: Richard Eckart de Castilho <re...@apache.org>
Date: Mon, 17 Aug 2020 at 15:37
Subject: Re: Maven builds out of control
To: <bu...@apache.org>
Dear Maven developers,
please fix or disable your CI builds ... other people need to use the
servers as well and in particular those who need to test jobs as part of
their job migration.
@INFRA: Maybe confine all Maven Jobs to a single specific slave for the
time being? That way, they only starve themselves.
Cheers,
-- Richard
--
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Fwd: Maven builds out of control
Posted by Arnaud Héritier <ah...@gmail.com>.
No I agree, it's a long term solution and orthogonal on cleaning our
branches/builds and limiting our usages
If these are static agents the best we could do at the CI level is to
restrict the usage of our builds to a limited number of agents
https://docs.cloudbees.com/docs/cloudbees-ci/latest/traditional-secure-guide/folders-plus#_controlled_agents
On Mon, Aug 17, 2020 at 10:55 AM Olivier Lamy <ol...@apache.org> wrote:
> traditional slaves
> I have a goal to help moving to kubernetes but this will not fix this
> problem :)
>
>
> On Mon, 17 Aug 2020 at 16:46, Arnaud Héritier <ah...@gmail.com> wrote:
>
> > Thanks for the feedback Gavin, it makes a lot of sense to me.
> > The new infrastructure is a traditional one (ie not hosted on
> Kubernetes) ?
> >
> > On Mon, Aug 17, 2020 at 10:44 AM Gavin McDonald <gm...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <ol...@apache.org>
> wrote:
> > >
> > > > +infra
> > > >
> > > > On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <aheritier@gmail.com
> >
> > > > wrote:
> > > >
> > > >> If we are consuming too much resources couldn't we have a dedicated
> > > >> Jenkins
> > > >>
> > > >> controller and/or limit our agents capacities to be the one ones to
> be
> > > >>
> > > >> impacted by these problems instead of all the users or this Jenkins
> > > >>
> > > >> controller ?
> > > >>
> > > >> Beam, Cassandra, CouchDB and few more have a dedicated controller :
> > > >>
> > > >> https://jenkins-ccos.apache.org/job/masters/
> > > >>
> > > >> What are the rules to have a dedicated one ?
> > > >>
> > > >
> > > All of those projects got their own master as they are contributing
> their
> > > own nodes and not using the ASF provided ones
> > >
> > > HTH
> > >
> > >
> > > >
> > > >>
> > > >>
> > > >> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <
> michaelo@apache.org>
> > > >> wrote:
> > > >>
> > > >>
> > > >>
> > > >> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> > > >>
> > > >> > > Furthermore, we need to clean up unused branches.
> > > >>
> > > >> > > What about removing branches older than 1yo?
> > > >>
> > > >> > > Then older than 6months?
> > > >>
> > > >> >
> > > >>
> > > >> > I consider branches which have been merged obsolete, but there is
> no
> > > >>
> > > >> > hard rule that an unmerged branch is obsolete. It may prepresent
> an
> > > >>
> > > >> > unfished state or something which was not agreed on. Dropping work
> > > does
> > > >>
> > > >> > not feel right. It needs to be decided on a case-by-case approach.
> > > >>
> > > >> > I have done this recently for Wagon and Resolver.
> > > >>
> > > >> >
> > > >>
> > > >> > Michael
> > > >>
> > > >> >
> > > >>
> > > >> >
> > ---------------------------------------------------------------------
> > > >>
> > > >> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > > >>
> > > >> > For additional commands, e-mail: dev-help@maven.apache.org
> > > >>
> > > >> >
> > > >>
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Arnaud Héritier
> > > >>
> > > >> Twitter/Skype : aheritier
> > > >>
> > > >> --
> > > > Olivier Lamy
> > > > http://twitter.com/olamy | http://linkedin.com/in/olamy
> > > >
> > >
> > >
> > > --
> > >
> > > *Gavin McDonald*
> > > Systems Administrator
> > > ASF Infrastructure Team
> > >
> >
> >
> > --
> > Arnaud Héritier
> > Twitter/Skype : aheritier
> >
>
>
> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>
--
Arnaud Héritier
Twitter/Skype : aheritier
Re: Fwd: Maven builds out of control
Posted by Olivier Lamy <ol...@apache.org>.
traditional slaves
I have a goal to help moving to kubernetes but this will not fix this
problem :)
On Mon, 17 Aug 2020 at 16:46, Arnaud Héritier <ah...@gmail.com> wrote:
> Thanks for the feedback Gavin, it makes a lot of sense to me.
> The new infrastructure is a traditional one (ie not hosted on Kubernetes) ?
>
> On Mon, Aug 17, 2020 at 10:44 AM Gavin McDonald <gm...@apache.org>
> wrote:
>
> > Hi,
> >
> > On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <ol...@apache.org> wrote:
> >
> > > +infra
> > >
> > > On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <ah...@gmail.com>
> > > wrote:
> > >
> > >> If we are consuming too much resources couldn't we have a dedicated
> > >> Jenkins
> > >>
> > >> controller and/or limit our agents capacities to be the one ones to be
> > >>
> > >> impacted by these problems instead of all the users or this Jenkins
> > >>
> > >> controller ?
> > >>
> > >> Beam, Cassandra, CouchDB and few more have a dedicated controller :
> > >>
> > >> https://jenkins-ccos.apache.org/job/masters/
> > >>
> > >> What are the rules to have a dedicated one ?
> > >>
> > >
> > All of those projects got their own master as they are contributing their
> > own nodes and not using the ASF provided ones
> >
> > HTH
> >
> >
> > >
> > >>
> > >>
> > >> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org>
> > >> wrote:
> > >>
> > >>
> > >>
> > >> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> > >>
> > >> > > Furthermore, we need to clean up unused branches.
> > >>
> > >> > > What about removing branches older than 1yo?
> > >>
> > >> > > Then older than 6months?
> > >>
> > >> >
> > >>
> > >> > I consider branches which have been merged obsolete, but there is no
> > >>
> > >> > hard rule that an unmerged branch is obsolete. It may prepresent an
> > >>
> > >> > unfished state or something which was not agreed on. Dropping work
> > does
> > >>
> > >> > not feel right. It needs to be decided on a case-by-case approach.
> > >>
> > >> > I have done this recently for Wagon and Resolver.
> > >>
> > >> >
> > >>
> > >> > Michael
> > >>
> > >> >
> > >>
> > >> >
> ---------------------------------------------------------------------
> > >>
> > >> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> > >>
> > >> > For additional commands, e-mail: dev-help@maven.apache.org
> > >>
> > >> >
> > >>
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >>
> > >> Arnaud Héritier
> > >>
> > >> Twitter/Skype : aheritier
> > >>
> > >> --
> > > Olivier Lamy
> > > http://twitter.com/olamy | http://linkedin.com/in/olamy
> > >
> >
> >
> > --
> >
> > *Gavin McDonald*
> > Systems Administrator
> > ASF Infrastructure Team
> >
>
>
> --
> Arnaud Héritier
> Twitter/Skype : aheritier
>
--
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Fwd: Maven builds out of control
Posted by Arnaud Héritier <ah...@gmail.com>.
Thanks for the feedback Gavin, it makes a lot of sense to me.
The new infrastructure is a traditional one (ie not hosted on Kubernetes) ?
On Mon, Aug 17, 2020 at 10:44 AM Gavin McDonald <gm...@apache.org>
wrote:
> Hi,
>
> On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <ol...@apache.org> wrote:
>
> > +infra
> >
> > On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <ah...@gmail.com>
> > wrote:
> >
> >> If we are consuming too much resources couldn't we have a dedicated
> >> Jenkins
> >>
> >> controller and/or limit our agents capacities to be the one ones to be
> >>
> >> impacted by these problems instead of all the users or this Jenkins
> >>
> >> controller ?
> >>
> >> Beam, Cassandra, CouchDB and few more have a dedicated controller :
> >>
> >> https://jenkins-ccos.apache.org/job/masters/
> >>
> >> What are the rules to have a dedicated one ?
> >>
> >
> All of those projects got their own master as they are contributing their
> own nodes and not using the ASF provided ones
>
> HTH
>
>
> >
> >>
> >>
> >> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org>
> >> wrote:
> >>
> >>
> >>
> >> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> >>
> >> > > Furthermore, we need to clean up unused branches.
> >>
> >> > > What about removing branches older than 1yo?
> >>
> >> > > Then older than 6months?
> >>
> >> >
> >>
> >> > I consider branches which have been merged obsolete, but there is no
> >>
> >> > hard rule that an unmerged branch is obsolete. It may prepresent an
> >>
> >> > unfished state or something which was not agreed on. Dropping work
> does
> >>
> >> > not feel right. It needs to be decided on a case-by-case approach.
> >>
> >> > I have done this recently for Wagon and Resolver.
> >>
> >> >
> >>
> >> > Michael
> >>
> >> >
> >>
> >> > ---------------------------------------------------------------------
> >>
> >> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> >>
> >> > For additional commands, e-mail: dev-help@maven.apache.org
> >>
> >> >
> >>
> >> >
> >>
> >>
> >>
> >> --
> >>
> >> Arnaud Héritier
> >>
> >> Twitter/Skype : aheritier
> >>
> >> --
> > Olivier Lamy
> > http://twitter.com/olamy | http://linkedin.com/in/olamy
> >
>
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
>
--
Arnaud Héritier
Twitter/Skype : aheritier
Re: Fwd: Maven builds out of control
Posted by Gavin McDonald <gm...@apache.org>.
Hi,
On Mon, Aug 17, 2020 at 10:35 AM Olivier Lamy <ol...@apache.org> wrote:
> +infra
>
> On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <ah...@gmail.com>
> wrote:
>
>> If we are consuming too much resources couldn't we have a dedicated
>> Jenkins
>>
>> controller and/or limit our agents capacities to be the one ones to be
>>
>> impacted by these problems instead of all the users or this Jenkins
>>
>> controller ?
>>
>> Beam, Cassandra, CouchDB and few more have a dedicated controller :
>>
>> https://jenkins-ccos.apache.org/job/masters/
>>
>> What are the rules to have a dedicated one ?
>>
>
All of those projects got their own master as they are contributing their
own nodes and not using the ASF provided ones
HTH
>
>>
>>
>> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org>
>> wrote:
>>
>>
>>
>> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
>>
>> > > Furthermore, we need to clean up unused branches.
>>
>> > > What about removing branches older than 1yo?
>>
>> > > Then older than 6months?
>>
>> >
>>
>> > I consider branches which have been merged obsolete, but there is no
>>
>> > hard rule that an unmerged branch is obsolete. It may prepresent an
>>
>> > unfished state or something which was not agreed on. Dropping work does
>>
>> > not feel right. It needs to be decided on a case-by-case approach.
>>
>> > I have done this recently for Wagon and Resolver.
>>
>> >
>>
>> > Michael
>>
>> >
>>
>> > ---------------------------------------------------------------------
>>
>> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>>
>> > For additional commands, e-mail: dev-help@maven.apache.org
>>
>> >
>>
>> >
>>
>>
>>
>> --
>>
>> Arnaud Héritier
>>
>> Twitter/Skype : aheritier
>>
>> --
> Olivier Lamy
> http://twitter.com/olamy | http://linkedin.com/in/olamy
>
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Re: Fwd: Maven builds out of control
Posted by Olivier Lamy <ol...@apache.org>.
+infra
On Mon, 17 Aug 2020 at 4:23 pm, Arnaud Héritier <ah...@gmail.com> wrote:
> If we are consuming too much resources couldn't we have a dedicated Jenkins
>
> controller and/or limit our agents capacities to be the one ones to be
>
> impacted by these problems instead of all the users or this Jenkins
>
> controller ?
>
> Beam, Cassandra, CouchDB and few more have a dedicated controller :
>
> https://jenkins-ccos.apache.org/job/masters/
>
> What are the rules to have a dedicated one ?
>
>
>
> On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org>
> wrote:
>
>
>
> > Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
>
> > > Furthermore, we need to clean up unused branches.
>
> > > What about removing branches older than 1yo?
>
> > > Then older than 6months?
>
> >
>
> > I consider branches which have been merged obsolete, but there is no
>
> > hard rule that an unmerged branch is obsolete. It may prepresent an
>
> > unfished state or something which was not agreed on. Dropping work does
>
> > not feel right. It needs to be decided on a case-by-case approach.
>
> > I have done this recently for Wagon and Resolver.
>
> >
>
> > Michael
>
> >
>
> > ---------------------------------------------------------------------
>
> > To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
>
> > For additional commands, e-mail: dev-help@maven.apache.org
>
> >
>
> >
>
>
>
> --
>
> Arnaud Héritier
>
> Twitter/Skype : aheritier
>
> --
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Fwd: Maven builds out of control
Posted by Arnaud Héritier <ah...@gmail.com>.
If we are consuming too much resources couldn't we have a dedicated Jenkins
controller and/or limit our agents capacities to be the one ones to be
impacted by these problems instead of all the users or this Jenkins
controller ?
Beam, Cassandra, CouchDB and few more have a dedicated controller :
https://jenkins-ccos.apache.org/job/masters/
What are the rules to have a dedicated one ?
On Mon, Aug 17, 2020 at 10:02 AM Michael Osipov <mi...@apache.org> wrote:
> Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> > Furthermore, we need to clean up unused branches.
> > What about removing branches older than 1yo?
> > Then older than 6months?
>
> I consider branches which have been merged obsolete, but there is no
> hard rule that an unmerged branch is obsolete. It may prepresent an
> unfished state or something which was not agreed on. Dropping work does
> not feel right. It needs to be decided on a case-by-case approach.
> I have done this recently for Wagon and Resolver.
>
> Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
Arnaud Héritier
Twitter/Skype : aheritier
Re: Fwd: Maven builds out of control
Posted by Michael Osipov <mi...@apache.org>.
Am 2020-08-17 um 09:55 schrieb Olivier Lamy:
> Furthermore, we need to clean up unused branches.
> What about removing branches older than 1yo?
> Then older than 6months?
I consider branches which have been merged obsolete, but there is no
hard rule that an unmerged branch is obsolete. It may prepresent an
unfished state or something which was not agreed on. Dropping work does
not feel right. It needs to be decided on a case-by-case approach.
I have done this recently for Wagon and Resolver.
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org
Re: Fwd: Maven builds out of control
Posted by Olivier Lamy <ol...@apache.org>.
Yes, we can limit branches using regex.
if use @dependabot merge the branch is deleted
On Mon, 17 Aug 2020 at 16:41, Maarten Mulders <mt...@apache.org> wrote:
> On 17/08/2020 09:55, Olivier Lamy wrote:
> > Hi there
> > we definitely need to reduce the matrix we have and cleanup branches ...
> > so I reduced the matrix (no more 14,15)
> > and only '3.3.x','3.6.x'
> >
> > Furthermore, we need to clean up unused branches.
> > What about removing branches older than 1yo?
> > Then older than 6months?
>
> Nowadays there are a lot of branches created by Dependabot.
> I think it's good to update dependencies more frequently than we
> currently do. But we shouldn't be building all those branches on Jenkins
> immediately, and without (human) review. There's a risk of unreviewed
> code being ran on Jenkins. Also, as we can see from this situation, it
> creates so many branches that are never cleaned up.
>
> If there would be a way to prevent those branches being built, that
> would definitely save space in the build queue. All branches start with
> dependabot/, but I haven't found a way yet to filter those out.
>
> Thanks,
>
> Maarten
>
> > ---------- Forwarded message ---------
> > From: Richard Eckart de Castilho <re...@apache.org>
> > Date: Mon, 17 Aug 2020 at 15:37
> > Subject: Re: Maven builds out of control
> > To: <bu...@apache.org>
> >
> >
> > Dear Maven developers,
> >
> > please fix or disable your CI builds ... other people need to use the
> > servers as well and in particular those who need to test jobs as part of
> > their job migration.
> >
> > @INFRA: Maybe confine all Maven Jobs to a single specific slave for the
> > time being? That way, they only starve themselves.
> >
> > Cheers,
> >
> > -- Richard
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
> For additional commands, e-mail: dev-help@maven.apache.org
>
>
--
Olivier Lamy
http://twitter.com/olamy | http://linkedin.com/in/olamy
Re: Fwd: Maven builds out of control
Posted by Maarten Mulders <mt...@apache.org>.
On 17/08/2020 09:55, Olivier Lamy wrote:
> Hi there
> we definitely need to reduce the matrix we have and cleanup branches ...
> so I reduced the matrix (no more 14,15)
> and only '3.3.x','3.6.x'
>
> Furthermore, we need to clean up unused branches.
> What about removing branches older than 1yo?
> Then older than 6months?
Nowadays there are a lot of branches created by Dependabot.
I think it's good to update dependencies more frequently than we
currently do. But we shouldn't be building all those branches on Jenkins
immediately, and without (human) review. There's a risk of unreviewed
code being ran on Jenkins. Also, as we can see from this situation, it
creates so many branches that are never cleaned up.
If there would be a way to prevent those branches being built, that
would definitely save space in the build queue. All branches start with
dependabot/, but I haven't found a way yet to filter those out.
Thanks,
Maarten
> ---------- Forwarded message ---------
> From: Richard Eckart de Castilho <re...@apache.org>
> Date: Mon, 17 Aug 2020 at 15:37
> Subject: Re: Maven builds out of control
> To: <bu...@apache.org>
>
>
> Dear Maven developers,
>
> please fix or disable your CI builds ... other people need to use the
> servers as well and in particular those who need to test jobs as part of
> their job migration.
>
> @INFRA: Maybe confine all Maven Jobs to a single specific slave for the
> time being? That way, they only starve themselves.
>
> Cheers,
>
> -- Richard
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@maven.apache.org
For additional commands, e-mail: dev-help@maven.apache.org