You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Gavin McDonald <gm...@apache.org> on 2021/11/01 13:13:29 UTC
Next builds@ meeting
Hi All,
I think it is past time to have another meeting.
The stresses and strains of our current CI makes me think we should have
another discussion.
Link to the cwiki page:
https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
If this week doesn't work then we can postpone until next week.
HTH
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Fwd: Next builds@ meeting
Posted by Zoran Regvart <zo...@regvart.com>.
FYI there will be a meeting around builds at ASF, perhaps there we
have issues worth discussing there.
See also:
https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-11
zoran
---------- Forwarded message ---------
From: Gavin McDonald <gm...@apache.org>
Date: Tue, Nov 9, 2021 at 9:15 AM
Subject: Re: Next builds@ meeting
To: builds <bu...@apache.org>
Reminder to all that the builds@ meeting will take place this week, in 2
days.
Hope you can all make it!
Gav...
On Thu, Nov 4, 2021 at 8:58 AM Gavin McDonald <gm...@apache.org> wrote:
> Hi All.
>
> It seems next week is better for some so I'll push this out to the 11th.
>
> HTH
>
> On Mon, Nov 1, 2021 at 4:08 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
>> I'm interested in participating. The Heron project is hitting limits
>> with TravisCI, I'd like to find a way to make our process better whether
>> it's with Jenkins/GA or something else. Next week would be better for me,
>> if we can push it back.
>>
>> On Mon, Nov 1, 2021 at 8:49 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> I updated the graphs. I think things have changed a lot since December
>>> last
>>> year.
>>>
>>> There are no absurd-high (and way above the limits) peaks of workflows in
>>> progress for some projects (e.g. Pulsar). However a number of apache
>>> repos
>>> using GA grows steadily. Number of workflows "queued" fluctuates for many
>>> projects and for some projects (spark) it seems it steadily grows.
>>>
>>> Observation from the Airflow side (and I think lack of "escalation"
>>> indicates that other projects have the same) is that we stopped observing
>>> prolonged periods where a lot of jobs are queued for a long time (many
>>> hours). One thing is that we moved a lot of the workload to our
>>> self-hosted
>>> runners - when it comes to public runners, there are short peaks of
>>> queued
>>> jobs but they seem to catch-up relatively quickly (max few hours).
>>>
>>> When I see those graph, my interpretation is that Github got a lot better
>>> queuing algorithms.Previously it seemed to be possible by a single
>>> project
>>> to take over the queue and get the number of "in progress" way higher
>>> than
>>> the 150 jobs limit, and I think that effectively blocked other projects
>>> for
>>> a long time. Currently it seems that the algorithm they have balances out
>>> the workload between projects much better - and if there are projects
>>> that
>>> have more and more workloads, it impacts their own queue size much more
>>> than queuing time of other projects - see Spark's average number of
>>> queued
>>> workflows.
>>>
>>> The graphs IMHO are much healthier than they were a year ago.
>>>
>>> I wonder what is the perception of others :).
>>>
>>> BTW. It would be great if we could move it by a week - this week is
>>> shorter
>>> for many and some have longer holidays (1st of Nov is a holiday in a
>>> number
>>> of countries).
>>>
>>> On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <gm...@apache.org>
>>> wrote:
>>>
>>> > Hi All,
>>> >
>>> > I think it is past time to have another meeting.
>>> > The stresses and strains of our current CI makes me think we should
>>> have
>>> > another discussion.
>>> >
>>> > Link to the cwiki page:
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
>>> >
>>> > If this week doesn't work then we can postpone until next week.
>>> >
>>> > HTH
>>> >
>>> > --
>>> >
>>> > *Gavin McDonald*
>>> > Systems Administrator
>>> > ASF Infrastructure Team
>>> >
>>>
>>
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
>
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
--
Zoran Regvart
Re: Next builds@ meeting
Posted by Martin Grigorov <mg...@apache.org>.
Hi,
About Oracle Cloud: it seems the beefy ARM64 machine was available only for
the trial period.
The Always Free tier allows up to 4 CPU (@2.8GHz Ampere Altra 80), 24 GB of
RAM, 45GB disk and 4 Gbps network bandwidth. One can choose between Oracle
EL, Ubuntu and CentOS - several versions of each.
For x86_64 VMs the Free tier allows up to 2 VMs with 1 CPU (2.0 GHz AMD
EPYC™ 7551 (Naples)), 1GB of RAM and 0.48 Gbps bandwidth. Same choice for
OSes.
I've just setup self-hosted runner on ARM64 VM with 1 CPU and 6GB RAM. The
Apache Avro Rust GHA workflows are around 70% slower than the public GHA
runners:
- public GHA runner builds Avro in 3 mins
- the self-hosted runner in 5 mins
More info at:
https://docs.oracle.com/en-us/iaas/Content/FreeTier/freetier_topic-Always_Free_Resources.htm#resources
Soon I will try the modified GHA runner by Apache Airflow.
Once I see it working I will write a blog about all this.
Regards,
Martin
On Tue, Nov 9, 2021 at 10:55 PM Craig Hunt <ma...@craighunt.us> wrote:
> Great, thanks!
>
> On Tue, Nov 9, 2021 at 12:46 PM Gavin McDonald <gm...@apache.org>
> wrote:
> >
> > Hi Craig,
> >
> > This meeting (and most builds meetings) is open to all, I have added
> > you to the guest list.
> >
> > Keep an eye out on Thursday where I'll add the meeting id to the builds@
> > list
> > and the wiki page.
> >
> >
> >
> > On Tue, Nov 9, 2021 at 6:12 PM Craig Hunt <ma...@craighunt.us> wrote:
> >
> > > Gavin, I am a new potential volunteer with a strong interest in ASF
> > > GitBox CI/CD build operations and would like to attend your meeting
> > > strictly as an observer (knowledge gain exercise).
> > > Greg Stein confirmed by email that he thought it may be OK for me to
> > > attend if the meeting was public.
> > > However, I am not clear if the meeting is public, as I first thought,
> > > if it is can I be added as a guest observer? I cannot add myself on
> > > Confluence it seems.
> > > I know there are good reasons for security now and I have NO essential
> > > need as a potential volunteer to attend, so please let me know your
> > > policy preference.
> > > Best regards, Craig Richard Hunt
> > >
> > >
> > > On Tue, Nov 9, 2021 at 2:15 AM Gavin McDonald <gm...@apache.org>
> > > wrote:
> > > >
> > > > Reminder to all that the builds@ meeting will take place this week,
> in 2
> > > > days.
> > > >
> > > > Hope you can all make it!
> > > >
> > > > Gav...
> > > >
> > > >
> > > > On Thu, Nov 4, 2021 at 8:58 AM Gavin McDonald <gm...@apache.org>
> > > wrote:
> > > >
> > > > > Hi All.
> > > > >
> > > > > It seems next week is better for some so I'll push this out to the
> > > 11th.
> > > > >
> > > > > HTH
> > > > >
> > > > > On Mon, Nov 1, 2021 at 4:08 PM Josh Fischer <jo...@joshfischer.io>
> > > wrote:
> > > > >
> > > > >> I'm interested in participating. The Heron project is hitting
> limits
> > > > >> with TravisCI, I'd like to find a way to make our process better
> > > whether
> > > > >> it's with Jenkins/GA or something else. Next week would be better
> for
> > > me,
> > > > >> if we can push it back.
> > > > >>
> > > > >> On Mon, Nov 1, 2021 at 8:49 AM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > > >>
> > > > >>> I updated the graphs. I think things have changed a lot since
> > > December
> > > > >>> last
> > > > >>> year.
> > > > >>>
> > > > >>> There are no absurd-high (and way above the limits) peaks of
> > > workflows in
> > > > >>> progress for some projects (e.g. Pulsar). However a number of
> apache
> > > > >>> repos
> > > > >>> using GA grows steadily. Number of workflows "queued" fluctuates
> for
> > > many
> > > > >>> projects and for some projects (spark) it seems it steadily
> grows.
> > > > >>>
> > > > >>> Observation from the Airflow side (and I think lack of
> "escalation"
> > > > >>> indicates that other projects have the same) is that we stopped
> > > observing
> > > > >>> prolonged periods where a lot of jobs are queued for a long time
> > > (many
> > > > >>> hours). One thing is that we moved a lot of the workload to our
> > > > >>> self-hosted
> > > > >>> runners - when it comes to public runners, there are short peaks
> of
> > > > >>> queued
> > > > >>> jobs but they seem to catch-up relatively quickly (max few
> hours).
> > > > >>>
> > > > >>> When I see those graph, my interpretation is that Github got a
> lot
> > > better
> > > > >>> queuing algorithms.Previously it seemed to be possible by a
> single
> > > > >>> project
> > > > >>> to take over the queue and get the number of "in progress" way
> higher
> > > > >>> than
> > > > >>> the 150 jobs limit, and I think that effectively blocked other
> > > projects
> > > > >>> for
> > > > >>> a long time. Currently it seems that the algorithm they have
> > > balances out
> > > > >>> the workload between projects much better - and if there are
> projects
> > > > >>> that
> > > > >>> have more and more workloads, it impacts their own queue size
> much
> > > more
> > > > >>> than queuing time of other projects - see Spark's average number
> of
> > > > >>> queued
> > > > >>> workflows.
> > > > >>>
> > > > >>> The graphs IMHO are much healthier than they were a year ago.
> > > > >>>
> > > > >>> I wonder what is the perception of others :).
> > > > >>>
> > > > >>> BTW. It would be great if we could move it by a week - this week
> is
> > > > >>> shorter
> > > > >>> for many and some have longer holidays (1st of Nov is a holiday
> in a
> > > > >>> number
> > > > >>> of countries).
> > > > >>>
> > > > >>> On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <
> gmcdonald@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Hi All,
> > > > >>> >
> > > > >>> > I think it is past time to have another meeting.
> > > > >>> > The stresses and strains of our current CI makes me think we
> should
> > > > >>> have
> > > > >>> > another discussion.
> > > > >>> >
> > > > >>> > Link to the cwiki page:
> > > > >>> >
> > > > >>> >
> > > > >>>
> > >
> https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
> > > > >>> >
> > > > >>> > If this week doesn't work then we can postpone until next week.
> > > > >>> >
> > > > >>> > HTH
> > > > >>> >
> > > > >>> > --
> > > > >>> >
> > > > >>> > *Gavin McDonald*
> > > > >>> > Systems Administrator
> > > > >>> > ASF Infrastructure Team
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >
> > > > > --
> > > > >
> > > > > *Gavin McDonald*
> > > > > Systems Administrator
> > > > > ASF Infrastructure Team
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > *Gavin McDonald*
> > > > Systems Administrator
> > > > ASF Infrastructure Team
> > >
> >
> >
> > --
> >
> > *Gavin McDonald*
> > Systems Administrator
> > ASF Infrastructure Team
>
Re: Next builds@ meeting
Posted by Craig Hunt <ma...@craighunt.us>.
Great, thanks!
On Tue, Nov 9, 2021 at 12:46 PM Gavin McDonald <gm...@apache.org> wrote:
>
> Hi Craig,
>
> This meeting (and most builds meetings) is open to all, I have added
> you to the guest list.
>
> Keep an eye out on Thursday where I'll add the meeting id to the builds@
> list
> and the wiki page.
>
>
>
> On Tue, Nov 9, 2021 at 6:12 PM Craig Hunt <ma...@craighunt.us> wrote:
>
> > Gavin, I am a new potential volunteer with a strong interest in ASF
> > GitBox CI/CD build operations and would like to attend your meeting
> > strictly as an observer (knowledge gain exercise).
> > Greg Stein confirmed by email that he thought it may be OK for me to
> > attend if the meeting was public.
> > However, I am not clear if the meeting is public, as I first thought,
> > if it is can I be added as a guest observer? I cannot add myself on
> > Confluence it seems.
> > I know there are good reasons for security now and I have NO essential
> > need as a potential volunteer to attend, so please let me know your
> > policy preference.
> > Best regards, Craig Richard Hunt
> >
> >
> > On Tue, Nov 9, 2021 at 2:15 AM Gavin McDonald <gm...@apache.org>
> > wrote:
> > >
> > > Reminder to all that the builds@ meeting will take place this week, in 2
> > > days.
> > >
> > > Hope you can all make it!
> > >
> > > Gav...
> > >
> > >
> > > On Thu, Nov 4, 2021 at 8:58 AM Gavin McDonald <gm...@apache.org>
> > wrote:
> > >
> > > > Hi All.
> > > >
> > > > It seems next week is better for some so I'll push this out to the
> > 11th.
> > > >
> > > > HTH
> > > >
> > > > On Mon, Nov 1, 2021 at 4:08 PM Josh Fischer <jo...@joshfischer.io>
> > wrote:
> > > >
> > > >> I'm interested in participating. The Heron project is hitting limits
> > > >> with TravisCI, I'd like to find a way to make our process better
> > whether
> > > >> it's with Jenkins/GA or something else. Next week would be better for
> > me,
> > > >> if we can push it back.
> > > >>
> > > >> On Mon, Nov 1, 2021 at 8:49 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >>
> > > >>> I updated the graphs. I think things have changed a lot since
> > December
> > > >>> last
> > > >>> year.
> > > >>>
> > > >>> There are no absurd-high (and way above the limits) peaks of
> > workflows in
> > > >>> progress for some projects (e.g. Pulsar). However a number of apache
> > > >>> repos
> > > >>> using GA grows steadily. Number of workflows "queued" fluctuates for
> > many
> > > >>> projects and for some projects (spark) it seems it steadily grows.
> > > >>>
> > > >>> Observation from the Airflow side (and I think lack of "escalation"
> > > >>> indicates that other projects have the same) is that we stopped
> > observing
> > > >>> prolonged periods where a lot of jobs are queued for a long time
> > (many
> > > >>> hours). One thing is that we moved a lot of the workload to our
> > > >>> self-hosted
> > > >>> runners - when it comes to public runners, there are short peaks of
> > > >>> queued
> > > >>> jobs but they seem to catch-up relatively quickly (max few hours).
> > > >>>
> > > >>> When I see those graph, my interpretation is that Github got a lot
> > better
> > > >>> queuing algorithms.Previously it seemed to be possible by a single
> > > >>> project
> > > >>> to take over the queue and get the number of "in progress" way higher
> > > >>> than
> > > >>> the 150 jobs limit, and I think that effectively blocked other
> > projects
> > > >>> for
> > > >>> a long time. Currently it seems that the algorithm they have
> > balances out
> > > >>> the workload between projects much better - and if there are projects
> > > >>> that
> > > >>> have more and more workloads, it impacts their own queue size much
> > more
> > > >>> than queuing time of other projects - see Spark's average number of
> > > >>> queued
> > > >>> workflows.
> > > >>>
> > > >>> The graphs IMHO are much healthier than they were a year ago.
> > > >>>
> > > >>> I wonder what is the perception of others :).
> > > >>>
> > > >>> BTW. It would be great if we could move it by a week - this week is
> > > >>> shorter
> > > >>> for many and some have longer holidays (1st of Nov is a holiday in a
> > > >>> number
> > > >>> of countries).
> > > >>>
> > > >>> On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <gm...@apache.org>
> > > >>> wrote:
> > > >>>
> > > >>> > Hi All,
> > > >>> >
> > > >>> > I think it is past time to have another meeting.
> > > >>> > The stresses and strains of our current CI makes me think we should
> > > >>> have
> > > >>> > another discussion.
> > > >>> >
> > > >>> > Link to the cwiki page:
> > > >>> >
> > > >>> >
> > > >>>
> > https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
> > > >>> >
> > > >>> > If this week doesn't work then we can postpone until next week.
> > > >>> >
> > > >>> > HTH
> > > >>> >
> > > >>> > --
> > > >>> >
> > > >>> > *Gavin McDonald*
> > > >>> > Systems Administrator
> > > >>> > ASF Infrastructure Team
> > > >>> >
> > > >>>
> > > >>
> > > >
> > > > --
> > > >
> > > > *Gavin McDonald*
> > > > Systems Administrator
> > > > ASF Infrastructure Team
> > > >
> > >
> > >
> > > --
> > >
> > > *Gavin McDonald*
> > > Systems Administrator
> > > ASF Infrastructure Team
> >
>
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
Re: Next builds@ meeting
Posted by Gavin McDonald <gm...@apache.org>.
Hi Craig,
This meeting (and most builds meetings) is open to all, I have added
you to the guest list.
Keep an eye out on Thursday where I'll add the meeting id to the builds@
list
and the wiki page.
On Tue, Nov 9, 2021 at 6:12 PM Craig Hunt <ma...@craighunt.us> wrote:
> Gavin, I am a new potential volunteer with a strong interest in ASF
> GitBox CI/CD build operations and would like to attend your meeting
> strictly as an observer (knowledge gain exercise).
> Greg Stein confirmed by email that he thought it may be OK for me to
> attend if the meeting was public.
> However, I am not clear if the meeting is public, as I first thought,
> if it is can I be added as a guest observer? I cannot add myself on
> Confluence it seems.
> I know there are good reasons for security now and I have NO essential
> need as a potential volunteer to attend, so please let me know your
> policy preference.
> Best regards, Craig Richard Hunt
>
>
> On Tue, Nov 9, 2021 at 2:15 AM Gavin McDonald <gm...@apache.org>
> wrote:
> >
> > Reminder to all that the builds@ meeting will take place this week, in 2
> > days.
> >
> > Hope you can all make it!
> >
> > Gav...
> >
> >
> > On Thu, Nov 4, 2021 at 8:58 AM Gavin McDonald <gm...@apache.org>
> wrote:
> >
> > > Hi All.
> > >
> > > It seems next week is better for some so I'll push this out to the
> 11th.
> > >
> > > HTH
> > >
> > > On Mon, Nov 1, 2021 at 4:08 PM Josh Fischer <jo...@joshfischer.io>
> wrote:
> > >
> > >> I'm interested in participating. The Heron project is hitting limits
> > >> with TravisCI, I'd like to find a way to make our process better
> whether
> > >> it's with Jenkins/GA or something else. Next week would be better for
> me,
> > >> if we can push it back.
> > >>
> > >> On Mon, Nov 1, 2021 at 8:49 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >>
> > >>> I updated the graphs. I think things have changed a lot since
> December
> > >>> last
> > >>> year.
> > >>>
> > >>> There are no absurd-high (and way above the limits) peaks of
> workflows in
> > >>> progress for some projects (e.g. Pulsar). However a number of apache
> > >>> repos
> > >>> using GA grows steadily. Number of workflows "queued" fluctuates for
> many
> > >>> projects and for some projects (spark) it seems it steadily grows.
> > >>>
> > >>> Observation from the Airflow side (and I think lack of "escalation"
> > >>> indicates that other projects have the same) is that we stopped
> observing
> > >>> prolonged periods where a lot of jobs are queued for a long time
> (many
> > >>> hours). One thing is that we moved a lot of the workload to our
> > >>> self-hosted
> > >>> runners - when it comes to public runners, there are short peaks of
> > >>> queued
> > >>> jobs but they seem to catch-up relatively quickly (max few hours).
> > >>>
> > >>> When I see those graph, my interpretation is that Github got a lot
> better
> > >>> queuing algorithms.Previously it seemed to be possible by a single
> > >>> project
> > >>> to take over the queue and get the number of "in progress" way higher
> > >>> than
> > >>> the 150 jobs limit, and I think that effectively blocked other
> projects
> > >>> for
> > >>> a long time. Currently it seems that the algorithm they have
> balances out
> > >>> the workload between projects much better - and if there are projects
> > >>> that
> > >>> have more and more workloads, it impacts their own queue size much
> more
> > >>> than queuing time of other projects - see Spark's average number of
> > >>> queued
> > >>> workflows.
> > >>>
> > >>> The graphs IMHO are much healthier than they were a year ago.
> > >>>
> > >>> I wonder what is the perception of others :).
> > >>>
> > >>> BTW. It would be great if we could move it by a week - this week is
> > >>> shorter
> > >>> for many and some have longer holidays (1st of Nov is a holiday in a
> > >>> number
> > >>> of countries).
> > >>>
> > >>> On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <gm...@apache.org>
> > >>> wrote:
> > >>>
> > >>> > Hi All,
> > >>> >
> > >>> > I think it is past time to have another meeting.
> > >>> > The stresses and strains of our current CI makes me think we should
> > >>> have
> > >>> > another discussion.
> > >>> >
> > >>> > Link to the cwiki page:
> > >>> >
> > >>> >
> > >>>
> https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
> > >>> >
> > >>> > If this week doesn't work then we can postpone until next week.
> > >>> >
> > >>> > HTH
> > >>> >
> > >>> > --
> > >>> >
> > >>> > *Gavin McDonald*
> > >>> > Systems Administrator
> > >>> > ASF Infrastructure Team
> > >>> >
> > >>>
> > >>
> > >
> > > --
> > >
> > > *Gavin McDonald*
> > > Systems Administrator
> > > ASF Infrastructure Team
> > >
> >
> >
> > --
> >
> > *Gavin McDonald*
> > Systems Administrator
> > ASF Infrastructure Team
>
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Re: Next builds@ meeting
Posted by Craig Hunt <ma...@craighunt.us>.
Gavin, I am a new potential volunteer with a strong interest in ASF
GitBox CI/CD build operations and would like to attend your meeting
strictly as an observer (knowledge gain exercise).
Greg Stein confirmed by email that he thought it may be OK for me to
attend if the meeting was public.
However, I am not clear if the meeting is public, as I first thought,
if it is can I be added as a guest observer? I cannot add myself on
Confluence it seems.
I know there are good reasons for security now and I have NO essential
need as a potential volunteer to attend, so please let me know your
policy preference.
Best regards, Craig Richard Hunt
On Tue, Nov 9, 2021 at 2:15 AM Gavin McDonald <gm...@apache.org> wrote:
>
> Reminder to all that the builds@ meeting will take place this week, in 2
> days.
>
> Hope you can all make it!
>
> Gav...
>
>
> On Thu, Nov 4, 2021 at 8:58 AM Gavin McDonald <gm...@apache.org> wrote:
>
> > Hi All.
> >
> > It seems next week is better for some so I'll push this out to the 11th.
> >
> > HTH
> >
> > On Mon, Nov 1, 2021 at 4:08 PM Josh Fischer <jo...@joshfischer.io> wrote:
> >
> >> I'm interested in participating. The Heron project is hitting limits
> >> with TravisCI, I'd like to find a way to make our process better whether
> >> it's with Jenkins/GA or something else. Next week would be better for me,
> >> if we can push it back.
> >>
> >> On Mon, Nov 1, 2021 at 8:49 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> >>
> >>> I updated the graphs. I think things have changed a lot since December
> >>> last
> >>> year.
> >>>
> >>> There are no absurd-high (and way above the limits) peaks of workflows in
> >>> progress for some projects (e.g. Pulsar). However a number of apache
> >>> repos
> >>> using GA grows steadily. Number of workflows "queued" fluctuates for many
> >>> projects and for some projects (spark) it seems it steadily grows.
> >>>
> >>> Observation from the Airflow side (and I think lack of "escalation"
> >>> indicates that other projects have the same) is that we stopped observing
> >>> prolonged periods where a lot of jobs are queued for a long time (many
> >>> hours). One thing is that we moved a lot of the workload to our
> >>> self-hosted
> >>> runners - when it comes to public runners, there are short peaks of
> >>> queued
> >>> jobs but they seem to catch-up relatively quickly (max few hours).
> >>>
> >>> When I see those graph, my interpretation is that Github got a lot better
> >>> queuing algorithms.Previously it seemed to be possible by a single
> >>> project
> >>> to take over the queue and get the number of "in progress" way higher
> >>> than
> >>> the 150 jobs limit, and I think that effectively blocked other projects
> >>> for
> >>> a long time. Currently it seems that the algorithm they have balances out
> >>> the workload between projects much better - and if there are projects
> >>> that
> >>> have more and more workloads, it impacts their own queue size much more
> >>> than queuing time of other projects - see Spark's average number of
> >>> queued
> >>> workflows.
> >>>
> >>> The graphs IMHO are much healthier than they were a year ago.
> >>>
> >>> I wonder what is the perception of others :).
> >>>
> >>> BTW. It would be great if we could move it by a week - this week is
> >>> shorter
> >>> for many and some have longer holidays (1st of Nov is a holiday in a
> >>> number
> >>> of countries).
> >>>
> >>> On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <gm...@apache.org>
> >>> wrote:
> >>>
> >>> > Hi All,
> >>> >
> >>> > I think it is past time to have another meeting.
> >>> > The stresses and strains of our current CI makes me think we should
> >>> have
> >>> > another discussion.
> >>> >
> >>> > Link to the cwiki page:
> >>> >
> >>> >
> >>> https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
> >>> >
> >>> > If this week doesn't work then we can postpone until next week.
> >>> >
> >>> > HTH
> >>> >
> >>> > --
> >>> >
> >>> > *Gavin McDonald*
> >>> > Systems Administrator
> >>> > ASF Infrastructure Team
> >>> >
> >>>
> >>
> >
> > --
> >
> > *Gavin McDonald*
> > Systems Administrator
> > ASF Infrastructure Team
> >
>
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
Re: Next builds@ meeting
Posted by Gavin McDonald <gm...@apache.org>.
Reminder to all that the builds@ meeting will take place this week, in 2
days.
Hope you can all make it!
Gav...
On Thu, Nov 4, 2021 at 8:58 AM Gavin McDonald <gm...@apache.org> wrote:
> Hi All.
>
> It seems next week is better for some so I'll push this out to the 11th.
>
> HTH
>
> On Mon, Nov 1, 2021 at 4:08 PM Josh Fischer <jo...@joshfischer.io> wrote:
>
>> I'm interested in participating. The Heron project is hitting limits
>> with TravisCI, I'd like to find a way to make our process better whether
>> it's with Jenkins/GA or something else. Next week would be better for me,
>> if we can push it back.
>>
>> On Mon, Nov 1, 2021 at 8:49 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>>
>>> I updated the graphs. I think things have changed a lot since December
>>> last
>>> year.
>>>
>>> There are no absurd-high (and way above the limits) peaks of workflows in
>>> progress for some projects (e.g. Pulsar). However a number of apache
>>> repos
>>> using GA grows steadily. Number of workflows "queued" fluctuates for many
>>> projects and for some projects (spark) it seems it steadily grows.
>>>
>>> Observation from the Airflow side (and I think lack of "escalation"
>>> indicates that other projects have the same) is that we stopped observing
>>> prolonged periods where a lot of jobs are queued for a long time (many
>>> hours). One thing is that we moved a lot of the workload to our
>>> self-hosted
>>> runners - when it comes to public runners, there are short peaks of
>>> queued
>>> jobs but they seem to catch-up relatively quickly (max few hours).
>>>
>>> When I see those graph, my interpretation is that Github got a lot better
>>> queuing algorithms.Previously it seemed to be possible by a single
>>> project
>>> to take over the queue and get the number of "in progress" way higher
>>> than
>>> the 150 jobs limit, and I think that effectively blocked other projects
>>> for
>>> a long time. Currently it seems that the algorithm they have balances out
>>> the workload between projects much better - and if there are projects
>>> that
>>> have more and more workloads, it impacts their own queue size much more
>>> than queuing time of other projects - see Spark's average number of
>>> queued
>>> workflows.
>>>
>>> The graphs IMHO are much healthier than they were a year ago.
>>>
>>> I wonder what is the perception of others :).
>>>
>>> BTW. It would be great if we could move it by a week - this week is
>>> shorter
>>> for many and some have longer holidays (1st of Nov is a holiday in a
>>> number
>>> of countries).
>>>
>>> On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <gm...@apache.org>
>>> wrote:
>>>
>>> > Hi All,
>>> >
>>> > I think it is past time to have another meeting.
>>> > The stresses and strains of our current CI makes me think we should
>>> have
>>> > another discussion.
>>> >
>>> > Link to the cwiki page:
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
>>> >
>>> > If this week doesn't work then we can postpone until next week.
>>> >
>>> > HTH
>>> >
>>> > --
>>> >
>>> > *Gavin McDonald*
>>> > Systems Administrator
>>> > ASF Infrastructure Team
>>> >
>>>
>>
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
>
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Re: Next builds@ meeting
Posted by Gavin McDonald <gm...@apache.org>.
Hi All.
It seems next week is better for some so I'll push this out to the 11th.
HTH
On Mon, Nov 1, 2021 at 4:08 PM Josh Fischer <jo...@joshfischer.io> wrote:
> I'm interested in participating. The Heron project is hitting limits with
> TravisCI, I'd like to find a way to make our process better whether
> it's with Jenkins/GA or something else. Next week would be better for me,
> if we can push it back.
>
> On Mon, Nov 1, 2021 at 8:49 AM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> I updated the graphs. I think things have changed a lot since December
>> last
>> year.
>>
>> There are no absurd-high (and way above the limits) peaks of workflows in
>> progress for some projects (e.g. Pulsar). However a number of apache repos
>> using GA grows steadily. Number of workflows "queued" fluctuates for many
>> projects and for some projects (spark) it seems it steadily grows.
>>
>> Observation from the Airflow side (and I think lack of "escalation"
>> indicates that other projects have the same) is that we stopped observing
>> prolonged periods where a lot of jobs are queued for a long time (many
>> hours). One thing is that we moved a lot of the workload to our
>> self-hosted
>> runners - when it comes to public runners, there are short peaks of queued
>> jobs but they seem to catch-up relatively quickly (max few hours).
>>
>> When I see those graph, my interpretation is that Github got a lot better
>> queuing algorithms.Previously it seemed to be possible by a single project
>> to take over the queue and get the number of "in progress" way higher than
>> the 150 jobs limit, and I think that effectively blocked other projects
>> for
>> a long time. Currently it seems that the algorithm they have balances out
>> the workload between projects much better - and if there are projects that
>> have more and more workloads, it impacts their own queue size much more
>> than queuing time of other projects - see Spark's average number of queued
>> workflows.
>>
>> The graphs IMHO are much healthier than they were a year ago.
>>
>> I wonder what is the perception of others :).
>>
>> BTW. It would be great if we could move it by a week - this week is
>> shorter
>> for many and some have longer holidays (1st of Nov is a holiday in a
>> number
>> of countries).
>>
>> On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <gm...@apache.org>
>> wrote:
>>
>> > Hi All,
>> >
>> > I think it is past time to have another meeting.
>> > The stresses and strains of our current CI makes me think we should have
>> > another discussion.
>> >
>> > Link to the cwiki page:
>> >
>> >
>> https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
>> >
>> > If this week doesn't work then we can postpone until next week.
>> >
>> > HTH
>> >
>> > --
>> >
>> > *Gavin McDonald*
>> > Systems Administrator
>> > ASF Infrastructure Team
>> >
>>
>
--
*Gavin McDonald*
Systems Administrator
ASF Infrastructure Team
Re: Next builds@ meeting
Posted by Josh Fischer <jo...@joshfischer.io>.
I'm interested in participating. The Heron project is hitting limits with
TravisCI, I'd like to find a way to make our process better whether
it's with Jenkins/GA or something else. Next week would be better for me,
if we can push it back.
On Mon, Nov 1, 2021 at 8:49 AM Jarek Potiuk <ja...@potiuk.com> wrote:
> I updated the graphs. I think things have changed a lot since December last
> year.
>
> There are no absurd-high (and way above the limits) peaks of workflows in
> progress for some projects (e.g. Pulsar). However a number of apache repos
> using GA grows steadily. Number of workflows "queued" fluctuates for many
> projects and for some projects (spark) it seems it steadily grows.
>
> Observation from the Airflow side (and I think lack of "escalation"
> indicates that other projects have the same) is that we stopped observing
> prolonged periods where a lot of jobs are queued for a long time (many
> hours). One thing is that we moved a lot of the workload to our self-hosted
> runners - when it comes to public runners, there are short peaks of queued
> jobs but they seem to catch-up relatively quickly (max few hours).
>
> When I see those graph, my interpretation is that Github got a lot better
> queuing algorithms.Previously it seemed to be possible by a single project
> to take over the queue and get the number of "in progress" way higher than
> the 150 jobs limit, and I think that effectively blocked other projects for
> a long time. Currently it seems that the algorithm they have balances out
> the workload between projects much better - and if there are projects that
> have more and more workloads, it impacts their own queue size much more
> than queuing time of other projects - see Spark's average number of queued
> workflows.
>
> The graphs IMHO are much healthier than they were a year ago.
>
> I wonder what is the perception of others :).
>
> BTW. It would be great if we could move it by a week - this week is shorter
> for many and some have longer holidays (1st of Nov is a holiday in a number
> of countries).
>
> On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <gm...@apache.org>
> wrote:
>
> > Hi All,
> >
> > I think it is past time to have another meeting.
> > The stresses and strains of our current CI makes me think we should have
> > another discussion.
> >
> > Link to the cwiki page:
> >
> >
> https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
> >
> > If this week doesn't work then we can postpone until next week.
> >
> > HTH
> >
> > --
> >
> > *Gavin McDonald*
> > Systems Administrator
> > ASF Infrastructure Team
> >
>
Re: Next builds@ meeting
Posted by Jarek Potiuk <ja...@potiuk.com>.
I updated the graphs. I think things have changed a lot since December last
year.
There are no absurd-high (and way above the limits) peaks of workflows in
progress for some projects (e.g. Pulsar). However a number of apache repos
using GA grows steadily. Number of workflows "queued" fluctuates for many
projects and for some projects (spark) it seems it steadily grows.
Observation from the Airflow side (and I think lack of "escalation"
indicates that other projects have the same) is that we stopped observing
prolonged periods where a lot of jobs are queued for a long time (many
hours). One thing is that we moved a lot of the workload to our self-hosted
runners - when it comes to public runners, there are short peaks of queued
jobs but they seem to catch-up relatively quickly (max few hours).
When I see those graph, my interpretation is that Github got a lot better
queuing algorithms.Previously it seemed to be possible by a single project
to take over the queue and get the number of "in progress" way higher than
the 150 jobs limit, and I think that effectively blocked other projects for
a long time. Currently it seems that the algorithm they have balances out
the workload between projects much better - and if there are projects that
have more and more workloads, it impacts their own queue size much more
than queuing time of other projects - see Spark's average number of queued
workflows.
The graphs IMHO are much healthier than they were a year ago.
I wonder what is the perception of others :).
BTW. It would be great if we could move it by a week - this week is shorter
for many and some have longer holidays (1st of Nov is a holiday in a number
of countries).
On Mon, Nov 1, 2021 at 2:13 PM Gavin McDonald <gm...@apache.org> wrote:
> Hi All,
>
> I think it is past time to have another meeting.
> The stresses and strains of our current CI makes me think we should have
> another discussion.
>
> Link to the cwiki page:
>
> https://cwiki.apache.org/confluence/display/INFRA/Builds+Agenda+2021-11-04
>
> If this week doesn't work then we can postpone until next week.
>
> HTH
>
> --
>
> *Gavin McDonald*
> Systems Administrator
> ASF Infrastructure Team
>