You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Jarek Potiuk <ja...@potiuk.com> on 2026/10/22 17:16:35 UTC
[DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Hi everyone,
I hope you're all doing well! I've been looking at how Arrow uses
GitHub Actions. All ASF projects share the same pool of GitHub
runners, so every minute we save also shortens the queue for
everyone else. The good news: Arrow's CI is already in good shape
(path filters, concurrency cancelling, the "CI: Extra" labels,
RunsOn for CUDA). Even so, I found some easy wins, and I'd love to
hear what you think.
I was a listener at yesterday's board meeting and i understand
that Arrow has some CI "difficulties" and is looking for ways to
deal with and JBO was generally offering his overlook of that
(hence the CC:) but then there was a lot of buzz about the
#project-workflows-optimisations channel of ASF - and I just added
new "workflow-optimization" skill in Magpie that is
producing very good advice, and you can use it to implement changes.
those optimizations quickly.
Last week we used roughly 110-125k GitHub-hosted runner minutes.
Package Linux alone is about 40% of that.
Two things need attention soon:
1. The nightly failure reports have been silent since Oct 21.
report_ci.yml installs "python-version: 3", which now gives
Python 3.15. pygit2 has no wheel for 3.15 yet, so installing
Archery fails and no email or Zulip report goes out. Pinning
to 3.13 fixes it.
2. pull_request_target is being disabled for ASF repositories in
November 2026. dev_pr.yml, pr_bot.yml and pr_limit.yml use it.
PR Limit can simply become the .asf.yaml "creation_cap" setting,
with the same limit of 3 open PRs per contributor and no
workflow at all.
Ideas to save minutes:
- Stop running the Extra workflows (C++/R/CUDA Extra, Package Linux)
on every merge to main, since they already run every night. This
alone is about 25% of our weekly minutes, mostly Package Linux's
20 jobs of about 90 minutes each.
- Once a PR has a "CI: Extra" label, every push reruns the full
20-job Package Linux matrix. Running a smaller subset on PRs
(say amd64 only) and the full matrix nightly would save about
another 10%.
- Read the "CI: Extra" labels straight from the PR event instead
of starting 2 small jobs per Extra workflow (about 870 jobs a
week), and move the bot and labelling jobs to the new small
"ubuntu-slim" runners.
- Maybe: replace the full core run on each merge to main with a
daily scheduled "canary" run (about 5%).
- Housekeeping: max-parallel 15 on Package Linux and Python,
timeouts on the bot jobs, cron schedules off minute 0.
Altogether that's roughly 30%, or about 40% with the Package Linux
PR subset. I could open PRs from my Magpie skills, but I do not
know Arrow well enough to know which of those make sense,
so you might want to implement those changes yourself.
How I did it: I ran the new "workflow-optimization" skill from
Apache Magpie (https://magpie.apache.org/), in the repo-health
family of skills. It works with the AI agent you already use. It
reads a repository's workflows, .asf.yaml, Dependabot config and a
week of Actions run history, and checks them against Infra's
upcoming "Optimizing GitHub Actions workflows" guide (in review at
https://github.com/apache/infrastructure-website/pull/320). The
result is a prioritised plan where every finding points to a file
and line and the numbers behind it. The skill only reads; it
changes nothing in the repository.
You can try it yourself:
1. Follow the install guide:
https://github.com/apache/magpie/blob/main/docs/setup/marketplace-install.md
2. In Claude Code, install the skills:
/plugin install magpie-repo-health@apache-magpie
3. Run it in your Arrow checkout:
/magpie-repo-health:workflow-optimization
(or just ask your agent to "optimize our CI workflows")
When you run it with your agent, it ends by suggesting how to split
the findings into separate PRs, and offers to draft each one with
you, one at a time. You review everything before anything is pushed.
Jean-Baptiste, I've CC'd you since you're on both the Arrow and
Magpie sides. Feedback on the skill is very welcome!
Thanks, and have a great day!
Jarek
--
Analysis and draft prepared with help from Claude Code using the
Apache Magpie workflow-optimization skill; reviewed by me.
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Raúl Cumplido <ra...@apache.org>.
Thanks Jarek,
Lots of things to unpack thanks for the analysis.
One thing which seems wrong from the analysis, the report CI jobs are
currently working. Notifications are arriving to Zulip with nightly
reports just fine. I keep monitoring those daily.
El jue, 22 oct 2026 a las 20:46, Benson Muite
(<be...@emailplus.org>) escribió:
>
> Jarek Potiuk <ja...@potiuk.com> writes:
>
> Hi,
>
> > Hi everyone,
> >
> > I hope you're all doing well! I've been looking at how Arrow uses
> > GitHub Actions. All ASF projects share the same pool of GitHub
> > runners, so every minute we save also shortens the queue for
> > everyone else. The good news: Arrow's CI is already in good shape
> > (path filters, concurrency cancelling, the "CI: Extra" labels,
> > RunsOn for CUDA). Even so, I found some easy wins, and I'd love to
> > hear what you think.
> >
> > I was a listener at yesterday's board meeting and i understand
> > that Arrow has some CI "difficulties" and is looking for ways to
> > deal with and JBO was generally offering his overlook of that
> > (hence the CC:) but then there was a lot of buzz about the
> > #project-workflows-optimisations channel of ASF - and I just added
> > new "workflow-optimization" skill in Magpie that is
> > producing very good advice, and you can use it to implement changes.
> > those optimizations quickly.
> >
> > Last week we used roughly 110-125k GitHub-hosted runner minutes.
> > Package Linux alone is about 40% of that.
>
> Fedora does offer CI builds for packaging:
> https://packit.dev/
>
> Probably CentOS and other RPM builds could be moved to that.
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello Andrew,
Sorry for being far too harsh with you.
As you've seen we had some unexpected turn of events in Airflow, where,
unfortunately, one of our PMCs was not happy with the way I moved from
discussion to lazy consensus, I was afraid that "nudging" us would have
such an effect beforehand, but I had to deal with it regardless. So, for
the moment the "CI run from forks" waits for a final vote. Thanks for
supporting me there - and. I think the tension of the last few days and
"pressure" from several areas did not help me to be nice.
Really good job on the Datafusion and Arrow side. I'm really glad we all
understand what needs to be done to fight the "Tragedy of Commons" and are
working in that direction. This is really cool.
We have the shared optimization hints and the Magpie skill that applies
them. We already see good results from our self-hosted, RunsOn-based
switch, plus the CI from forks is almost completely working (and very
nicely so), although it introduces complexities in Airflow because we have
so many different types of tests. I think the pressure on the queue is
already much lower, and we will reduce it even more over the next days and
weeks. Then, we should be able to disseminate some of these learnings -
together, to other projects, maybe even help infrastructure build automated
monitoring and suggestions for fixes that can be shared with the PMCs? With
AI at our disposal - such tooling is usually a few prompts away.
I hope the Arrow dev community can take a look at further optimizations.
While working on Airflow, I have found quite non-obvious ones and, I would
even say, it's quite fun to look at, assess, analyse, and implement
them—and then see the results that everyone benefits from.
J.
On Fri, Oct 23, 2026 at 8:05 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> BTW. The PR I opened in the Infra page about workflow optimisation advices
> is up: https://infra.apache.org/github-actions-optimization.html - it's
> actually also used by Magpie skill to propose optimisations, but it's also
> a good idea to follow manually if someone does not want to use agents.
>
>
>
> On Fri, Oct 23, 2026 at 7:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>>
>>
>> On Fri, Oct 23, 2026 at 6:52 PM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>>> (also adding dev@datafusion as they are affected as well)
>>>
>>> The reason I am nudging is that for the last several days, contributors
>>> in arrow-rs and datafusion are waiting 3-7 hours for CI jobs to start. I
>>> have heard frustration from many community members. This seems to have
>>> gotten much worse in the last week. While waiting for jobs to start, I
>>> started investigating what projects were using the runners that our jobs
>>> were waiting behind (the script I use is attached to issue [2]) which lead
>>> me to Airflow. I am sorry if it seems I am singling that project out but it
>>> seems to be using significantly more than the next largest project I can
>>> find.
>>>
>>
>> As I explained you several times. We are well aware of it, and as I told
>> you - we are working on it. I explained and showed you the several weeks of
>> discussions we had on it anticipating it, and all the work I have done
>> convincing our PMC members that we need to invest in it. Continuous nudging
>> - with automated scripts (WTF!!!) is one of many ways to get the other PMC
>> members more touchy and angry. We were not only proactive and receptive,
>> acknowledging the issue weeks ago, but we also provided you with all the
>> information, including the plan we had already developed over several weeks
>> to make it happen and confirmed we are working on it—this includes Shahar's
>> dedication to working over his weekend to achieve it. Yes, it's Friday, yes
>> he is in Israel, and yes, you knew all that. Yet you still decided to nudge
>> people with automated (!) script.
>>
>> It's happening. It's progressing. If all you can do is write a nudgy
>> script then my answer is "get lost." You are not helping with that; you are
>> making things worse. I honestly now have a great urge to just stop
>> working—it's Friday evening for me and I also work way past my hours to
>> help. I could go to a party now. I am actually thinking about cancelling my
>> morning running local community event to improve things.
>>
>> But seeing that some people think that doing an automated nudge will make
>> their ego boosted and say "I made it happen" eventually. It's wrong.
>>
>> Not helping. Stop it please.
>>
>> Many projects are running their own workflows - and each of the project
>> should impleement optimizations to make things happen. I spent my time,
>> skills (literally) in Magpie and experience on workflow optimization to
>> propose a 30% - 40% improvement (see above) - that will help everyone, and
>> we can share that experience with others so they can use a similar approach.
>>
>> So my question is - which you did not answer - what YOU are doing to help
>> projects like Arrow that use a lot of resources (for a good reason of
>> course; Arrow is a friendly project and many of my friends are here). You
>> are part of Arrow as well Andrew? Can you help develop those optimizations
>> maybe for Airflow?
>>
>> I gave an easy recipe to follow. I know writing nudging scripts is easier
>> and you can take credit for "fixing things" because you nudged people who
>> absolutely do not need it.
>>
>> But my proposal is that we all roll up our sleeves and, rather than
>> pointing at others (who are already working night and day and complicating
>> their workflows enormously to get the 80% savings), we fix things in our
>> own garden first.
>>
>> I would really love that, Andrew.
>>
>> Love and peace - I mean no harm, it's just pretty annoying behaviour.
>>
>> J.
>>
>>
>>
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello Andrew,
Sorry for being far too harsh with you.
As you've seen we had some unexpected turn of events in Airflow, where,
unfortunately, one of our PMCs was not happy with the way I moved from
discussion to lazy consensus, I was afraid that "nudging" us would have
such an effect beforehand, but I had to deal with it regardless. So, for
the moment the "CI run from forks" waits for a final vote. Thanks for
supporting me there - and. I think the tension of the last few days and
"pressure" from several areas did not help me to be nice.
Really good job on the Datafusion and Arrow side. I'm really glad we all
understand what needs to be done to fight the "Tragedy of Commons" and are
working in that direction. This is really cool.
We have the shared optimization hints and the Magpie skill that applies
them. We already see good results from our self-hosted, RunsOn-based
switch, plus the CI from forks is almost completely working (and very
nicely so), although it introduces complexities in Airflow because we have
so many different types of tests. I think the pressure on the queue is
already much lower, and we will reduce it even more over the next days and
weeks. Then, we should be able to disseminate some of these learnings -
together, to other projects, maybe even help infrastructure build automated
monitoring and suggestions for fixes that can be shared with the PMCs? With
AI at our disposal - such tooling is usually a few prompts away.
I hope the Arrow dev community can take a look at further optimizations.
While working on Airflow, I have found quite non-obvious ones and, I would
even say, it's quite fun to look at, assess, analyse, and implement
them—and then see the results that everyone benefits from.
J.
On Fri, Oct 23, 2026 at 8:05 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> BTW. The PR I opened in the Infra page about workflow optimisation advices
> is up: https://infra.apache.org/github-actions-optimization.html - it's
> actually also used by Magpie skill to propose optimisations, but it's also
> a good idea to follow manually if someone does not want to use agents.
>
>
>
> On Fri, Oct 23, 2026 at 7:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>>
>>
>> On Fri, Oct 23, 2026 at 6:52 PM Andrew Lamb <an...@gmail.com>
>> wrote:
>>
>>> (also adding dev@datafusion as they are affected as well)
>>>
>>> The reason I am nudging is that for the last several days, contributors
>>> in arrow-rs and datafusion are waiting 3-7 hours for CI jobs to start. I
>>> have heard frustration from many community members. This seems to have
>>> gotten much worse in the last week. While waiting for jobs to start, I
>>> started investigating what projects were using the runners that our jobs
>>> were waiting behind (the script I use is attached to issue [2]) which lead
>>> me to Airflow. I am sorry if it seems I am singling that project out but it
>>> seems to be using significantly more than the next largest project I can
>>> find.
>>>
>>
>> As I explained you several times. We are well aware of it, and as I told
>> you - we are working on it. I explained and showed you the several weeks of
>> discussions we had on it anticipating it, and all the work I have done
>> convincing our PMC members that we need to invest in it. Continuous nudging
>> - with automated scripts (WTF!!!) is one of many ways to get the other PMC
>> members more touchy and angry. We were not only proactive and receptive,
>> acknowledging the issue weeks ago, but we also provided you with all the
>> information, including the plan we had already developed over several weeks
>> to make it happen and confirmed we are working on it—this includes Shahar's
>> dedication to working over his weekend to achieve it. Yes, it's Friday, yes
>> he is in Israel, and yes, you knew all that. Yet you still decided to nudge
>> people with automated (!) script.
>>
>> It's happening. It's progressing. If all you can do is write a nudgy
>> script then my answer is "get lost." You are not helping with that; you are
>> making things worse. I honestly now have a great urge to just stop
>> working—it's Friday evening for me and I also work way past my hours to
>> help. I could go to a party now. I am actually thinking about cancelling my
>> morning running local community event to improve things.
>>
>> But seeing that some people think that doing an automated nudge will make
>> their ego boosted and say "I made it happen" eventually. It's wrong.
>>
>> Not helping. Stop it please.
>>
>> Many projects are running their own workflows - and each of the project
>> should impleement optimizations to make things happen. I spent my time,
>> skills (literally) in Magpie and experience on workflow optimization to
>> propose a 30% - 40% improvement (see above) - that will help everyone, and
>> we can share that experience with others so they can use a similar approach.
>>
>> So my question is - which you did not answer - what YOU are doing to help
>> projects like Arrow that use a lot of resources (for a good reason of
>> course; Arrow is a friendly project and many of my friends are here). You
>> are part of Arrow as well Andrew? Can you help develop those optimizations
>> maybe for Airflow?
>>
>> I gave an easy recipe to follow. I know writing nudging scripts is easier
>> and you can take credit for "fixing things" because you nudged people who
>> absolutely do not need it.
>>
>> But my proposal is that we all roll up our sleeves and, rather than
>> pointing at others (who are already working night and day and complicating
>> their workflows enormously to get the 80% savings), we fix things in our
>> own garden first.
>>
>> I would really love that, Andrew.
>>
>> Love and peace - I mean no harm, it's just pretty annoying behaviour.
>>
>> J.
>>
>>
>>
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Jarek Potiuk <ja...@potiuk.com>.
BTW. The PR I opened in the Infra page about workflow optimisation advices
is up: https://infra.apache.org/github-actions-optimization.html - it's
actually also used by Magpie skill to propose optimisations, but it's also
a good idea to follow manually if someone does not want to use agents.
On Fri, Oct 23, 2026 at 7:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>
> On Fri, Oct 23, 2026 at 6:52 PM Andrew Lamb <an...@gmail.com>
> wrote:
>
>> (also adding dev@datafusion as they are affected as well)
>>
>> The reason I am nudging is that for the last several days, contributors
>> in arrow-rs and datafusion are waiting 3-7 hours for CI jobs to start. I
>> have heard frustration from many community members. This seems to have
>> gotten much worse in the last week. While waiting for jobs to start, I
>> started investigating what projects were using the runners that our jobs
>> were waiting behind (the script I use is attached to issue [2]) which lead
>> me to Airflow. I am sorry if it seems I am singling that project out but it
>> seems to be using significantly more than the next largest project I can
>> find.
>>
>
> As I explained you several times. We are well aware of it, and as I told
> you - we are working on it. I explained and showed you the several weeks of
> discussions we had on it anticipating it, and all the work I have done
> convincing our PMC members that we need to invest in it. Continuous nudging
> - with automated scripts (WTF!!!) is one of many ways to get the other PMC
> members more touchy and angry. We were not only proactive and receptive,
> acknowledging the issue weeks ago, but we also provided you with all the
> information, including the plan we had already developed over several weeks
> to make it happen and confirmed we are working on it—this includes Shahar's
> dedication to working over his weekend to achieve it. Yes, it's Friday, yes
> he is in Israel, and yes, you knew all that. Yet you still decided to nudge
> people with automated (!) script.
>
> It's happening. It's progressing. If all you can do is write a nudgy
> script then my answer is "get lost." You are not helping with that; you are
> making things worse. I honestly now have a great urge to just stop
> working—it's Friday evening for me and I also work way past my hours to
> help. I could go to a party now. I am actually thinking about cancelling my
> morning running local community event to improve things.
>
> But seeing that some people think that doing an automated nudge will make
> their ego boosted and say "I made it happen" eventually. It's wrong.
>
> Not helping. Stop it please.
>
> Many projects are running their own workflows - and each of the project
> should impleement optimizations to make things happen. I spent my time,
> skills (literally) in Magpie and experience on workflow optimization to
> propose a 30% - 40% improvement (see above) - that will help everyone, and
> we can share that experience with others so they can use a similar approach.
>
> So my question is - which you did not answer - what YOU are doing to help
> projects like Arrow that use a lot of resources (for a good reason of
> course; Arrow is a friendly project and many of my friends are here). You
> are part of Arrow as well Andrew? Can you help develop those optimizations
> maybe for Airflow?
>
> I gave an easy recipe to follow. I know writing nudging scripts is easier
> and you can take credit for "fixing things" because you nudged people who
> absolutely do not need it.
>
> But my proposal is that we all roll up our sleeves and, rather than
> pointing at others (who are already working night and day and complicating
> their workflows enormously to get the 80% savings), we fix things in our
> own garden first.
>
> I would really love that, Andrew.
>
> Love and peace - I mean no harm, it's just pretty annoying behaviour.
>
> J.
>
>
>
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Jarek Potiuk <ja...@potiuk.com>.
BTW. The PR I opened in the Infra page about workflow optimisation advices
is up: https://infra.apache.org/github-actions-optimization.html - it's
actually also used by Magpie skill to propose optimisations, but it's also
a good idea to follow manually if someone does not want to use agents.
On Fri, Oct 23, 2026 at 7:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>
> On Fri, Oct 23, 2026 at 6:52 PM Andrew Lamb <an...@gmail.com>
> wrote:
>
>> (also adding dev@datafusion as they are affected as well)
>>
>> The reason I am nudging is that for the last several days, contributors
>> in arrow-rs and datafusion are waiting 3-7 hours for CI jobs to start. I
>> have heard frustration from many community members. This seems to have
>> gotten much worse in the last week. While waiting for jobs to start, I
>> started investigating what projects were using the runners that our jobs
>> were waiting behind (the script I use is attached to issue [2]) which lead
>> me to Airflow. I am sorry if it seems I am singling that project out but it
>> seems to be using significantly more than the next largest project I can
>> find.
>>
>
> As I explained you several times. We are well aware of it, and as I told
> you - we are working on it. I explained and showed you the several weeks of
> discussions we had on it anticipating it, and all the work I have done
> convincing our PMC members that we need to invest in it. Continuous nudging
> - with automated scripts (WTF!!!) is one of many ways to get the other PMC
> members more touchy and angry. We were not only proactive and receptive,
> acknowledging the issue weeks ago, but we also provided you with all the
> information, including the plan we had already developed over several weeks
> to make it happen and confirmed we are working on it—this includes Shahar's
> dedication to working over his weekend to achieve it. Yes, it's Friday, yes
> he is in Israel, and yes, you knew all that. Yet you still decided to nudge
> people with automated (!) script.
>
> It's happening. It's progressing. If all you can do is write a nudgy
> script then my answer is "get lost." You are not helping with that; you are
> making things worse. I honestly now have a great urge to just stop
> working—it's Friday evening for me and I also work way past my hours to
> help. I could go to a party now. I am actually thinking about cancelling my
> morning running local community event to improve things.
>
> But seeing that some people think that doing an automated nudge will make
> their ego boosted and say "I made it happen" eventually. It's wrong.
>
> Not helping. Stop it please.
>
> Many projects are running their own workflows - and each of the project
> should impleement optimizations to make things happen. I spent my time,
> skills (literally) in Magpie and experience on workflow optimization to
> propose a 30% - 40% improvement (see above) - that will help everyone, and
> we can share that experience with others so they can use a similar approach.
>
> So my question is - which you did not answer - what YOU are doing to help
> projects like Arrow that use a lot of resources (for a good reason of
> course; Arrow is a friendly project and many of my friends are here). You
> are part of Arrow as well Andrew? Can you help develop those optimizations
> maybe for Airflow?
>
> I gave an easy recipe to follow. I know writing nudging scripts is easier
> and you can take credit for "fixing things" because you nudged people who
> absolutely do not need it.
>
> But my proposal is that we all roll up our sleeves and, rather than
> pointing at others (who are already working night and day and complicating
> their workflows enormously to get the 80% savings), we fix things in our
> own garden first.
>
> I would really love that, Andrew.
>
> Love and peace - I mean no harm, it's just pretty annoying behaviour.
>
> J.
>
>
>
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Jarek Potiuk <ja...@potiuk.com>.
On Fri, Oct 23, 2026 at 6:52 PM Andrew Lamb <an...@gmail.com> wrote:
> (also adding dev@datafusion as they are affected as well)
>
> The reason I am nudging is that for the last several days, contributors in
> arrow-rs and datafusion are waiting 3-7 hours for CI jobs to start. I have
> heard frustration from many community members. This seems to have
> gotten much worse in the last week. While waiting for jobs to start, I
> started investigating what projects were using the runners that our jobs
> were waiting behind (the script I use is attached to issue [2]) which lead
> me to Airflow. I am sorry if it seems I am singling that project out but it
> seems to be using significantly more than the next largest project I can
> find.
>
As I explained you several times. We are well aware of it, and as I told
you - we are working on it. I explained and showed you the several weeks of
discussions we had on it anticipating it, and all the work I have done
convincing our PMC members that we need to invest in it. Continuous nudging
- with automated scripts (WTF!!!) is one of many ways to get the other PMC
members more touchy and angry. We were not only proactive and receptive,
acknowledging the issue weeks ago, but we also provided you with all the
information, including the plan we had already developed over several weeks
to make it happen and confirmed we are working on it—this includes Shahar's
dedication to working over his weekend to achieve it. Yes, it's Friday, yes
he is in Israel, and yes, you knew all that. Yet you still decided to nudge
people with automated (!) script.
It's happening. It's progressing. If all you can do is write a nudgy script
then my answer is "get lost." You are not helping with that; you are making
things worse. I honestly now have a great urge to just stop working—it's
Friday evening for me and I also work way past my hours to help. I could go
to a party now. I am actually thinking about cancelling my morning running
local community event to improve things.
But seeing that some people think that doing an automated nudge will make
their ego boosted and say "I made it happen" eventually. It's wrong.
Not helping. Stop it please.
Many projects are running their own workflows - and each of the project
should impleement optimizations to make things happen. I spent my time,
skills (literally) in Magpie and experience on workflow optimization to
propose a 30% - 40% improvement (see above) - that will help everyone, and
we can share that experience with others so they can use a similar approach.
So my question is - which you did not answer - what YOU are doing to help
projects like Arrow that use a lot of resources (for a good reason of
course; Arrow is a friendly project and many of my friends are here). You
are part of Arrow as well Andrew? Can you help develop those optimizations
maybe for Airflow?
I gave an easy recipe to follow. I know writing nudging scripts is easier
and you can take credit for "fixing things" because you nudged people who
absolutely do not need it.
But my proposal is that we all roll up our sleeves and, rather than
pointing at others (who are already working night and day and complicating
their workflows enormously to get the 80% savings), we fix things in our
own garden first.
I would really love that, Andrew.
Love and peace - I mean no harm, it's just pretty annoying behaviour.
J.
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Jarek Potiuk <ja...@potiuk.com>.
On Fri, Oct 23, 2026 at 6:52 PM Andrew Lamb <an...@gmail.com> wrote:
> (also adding dev@datafusion as they are affected as well)
>
> The reason I am nudging is that for the last several days, contributors in
> arrow-rs and datafusion are waiting 3-7 hours for CI jobs to start. I have
> heard frustration from many community members. This seems to have
> gotten much worse in the last week. While waiting for jobs to start, I
> started investigating what projects were using the runners that our jobs
> were waiting behind (the script I use is attached to issue [2]) which lead
> me to Airflow. I am sorry if it seems I am singling that project out but it
> seems to be using significantly more than the next largest project I can
> find.
>
As I explained you several times. We are well aware of it, and as I told
you - we are working on it. I explained and showed you the several weeks of
discussions we had on it anticipating it, and all the work I have done
convincing our PMC members that we need to invest in it. Continuous nudging
- with automated scripts (WTF!!!) is one of many ways to get the other PMC
members more touchy and angry. We were not only proactive and receptive,
acknowledging the issue weeks ago, but we also provided you with all the
information, including the plan we had already developed over several weeks
to make it happen and confirmed we are working on it—this includes Shahar's
dedication to working over his weekend to achieve it. Yes, it's Friday, yes
he is in Israel, and yes, you knew all that. Yet you still decided to nudge
people with automated (!) script.
It's happening. It's progressing. If all you can do is write a nudgy script
then my answer is "get lost." You are not helping with that; you are making
things worse. I honestly now have a great urge to just stop working—it's
Friday evening for me and I also work way past my hours to help. I could go
to a party now. I am actually thinking about cancelling my morning running
local community event to improve things.
But seeing that some people think that doing an automated nudge will make
their ego boosted and say "I made it happen" eventually. It's wrong.
Not helping. Stop it please.
Many projects are running their own workflows - and each of the project
should impleement optimizations to make things happen. I spent my time,
skills (literally) in Magpie and experience on workflow optimization to
propose a 30% - 40% improvement (see above) - that will help everyone, and
we can share that experience with others so they can use a similar approach.
So my question is - which you did not answer - what YOU are doing to help
projects like Arrow that use a lot of resources (for a good reason of
course; Arrow is a friendly project and many of my friends are here). You
are part of Arrow as well Andrew? Can you help develop those optimizations
maybe for Airflow?
I gave an easy recipe to follow. I know writing nudging scripts is easier
and you can take credit for "fixing things" because you nudged people who
absolutely do not need it.
But my proposal is that we all roll up our sleeves and, rather than
pointing at others (who are already working night and day and complicating
their workflows enormously to get the 80% savings), we fix things in our
own garden first.
I would really love that, Andrew.
Love and peace - I mean no harm, it's just pretty annoying behaviour.
J.
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Andrew Lamb <an...@gmail.com>.
(also adding dev@datafusion as they are affected as well)
Hi Jarek,
For others who are looking for more context related to this email, there is
an ASF slack thread[1] and issue[2] with more background and data.
> And we are explaining that for weeks, yet, for some reason we get nudges
from Andrew every day - even if he knows very well we are working on it.
The reason I am nudging is that for the last several days, contributors in
arrow-rs and datafusion are waiting 3-7 hours for CI jobs to start. I have
heard frustration from many community members. This seems to have
gotten much worse in the last week. While waiting for jobs to start, I
started investigating what projects were using the runners that our jobs
were waiting behind (the script I use is attached to issue [2]) which lead
me to Airflow. I am sorry if it seems I am singling that project out but it
seems to be using significantly more than the next largest project I can
find.
> (Airflow and Arrow) are two biggest users of the CI in ASF.
Can you please share the data that you used to come to this conclusion? My
numbers[3] suggest over the last 24 hours Airflow has used roughly 10x more
shared runner time (106 runner-day vs 16 runner-days) than
arrow/arrow-rs/arrow-go.
> Andrew, are you actually working with other Arrow team members on
applying those improvements I proposed (or other?). It seems a bit one sided
when you have all proactiveness from our side, yet we do not hear what you
are doing to help the ASF.
I am not sure what you proposed as I don't use or follow Airflow; I can
tell you that both arrow-rs and DataFusion communities have been busy
trying to reduce our shared runner usage, I have listed a few examples
below but you can find a bunch more if you search for recent merge PRs in
those repositories
Andrew
[1]: https://the-asf.slack.com/archives/CBX4TSBQ8/p1792520326623569
[2]: https://github.com/apache/datafusion/issues/25088
[3]:
https://github.com/apache/datafusion/issues/25088#issuecomment-6369916889
DataFusion: PRs to reduce shared runner load
- https://github.com/apache/datafusion/pull/25591 (Oct 20) Runs the Rust
workflow for fork PRs on the contributor's fork.
- https://github.com/apache/datafusion/pull/26574 (Oct 21) Moves the
circular dependency check onto the fork
- https://github.com/apache/datafusion/pull/26571 (Oct 21) Moves about a
dozen small jobs to ubuntu-slim.
Arrow (Rust): PRs to reduce shared runner load
- https://github.com/apache/arrow-rs/pull/11788: "Run fork PR test suites
in contributor forks".
- https://github.com/apache/arrow-rs/pull/11774: "ci: move several jobs to
ubuntu-slim runner".
- https://github.com/apache/arrow-rs/pull/11631: "Run Miri on a daily
schedule, require it before releases".
On Fri, Oct 23, 2026 at 12:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello here,
>
> I did not want to nudge anyone here, but since Andrew does it. Just
> adding Andrew - who keeps on nudging Airflow in issues - we are working in
> Airflow on implementing **way** more improvements than that - for several
> weeks already - and we are close to conclusion.
> And the only way to deal with queueing is to get big users (Airflow and
> Arrow) are two biggest users of the CI in ASF. In Airflow we are working on
> 80% cutting - via using self-hosted runners and via delegating CI to the
> contributor's public runners.
>
> And we are explaining that for weeks, yet, for some reason we get nudges
> from Andrew every day - even if he knows very well we are working on it.
>
> And - just to be sure - those 30% - 40% savings above are easily
> achievable (way easier than the 80% of Airflow) - and it is also absolutely
> necessary to remove the pressure from the queue.,
>
> Andrew,
>
> Andrew, are you actually working with other Arrow team members on applying
> those improvements I proposed (or other?). It seems a bit one sided when
> you have all proactiveness from our side, yet we do not hear what you are
> doing to help the ASF.
>
> I would really appreciate **some** answer here how are you working with
> others in the Arrow to solve the issues. Are there any plans to improve
> things ?
>
> And yes Andrew - I expect answer from you,
>
> J.
>
> On Thu, Oct 22, 2026 at 8:46 PM Benson Muite <be...@emailplus.org>
> wrote:
>
>> Jarek Potiuk <ja...@potiuk.com> writes:
>>
>> Hi,
>>
>> > Hi everyone,
>> >
>> > I hope you're all doing well! I've been looking at how Arrow uses
>> > GitHub Actions. All ASF projects share the same pool of GitHub
>> > runners, so every minute we save also shortens the queue for
>> > everyone else. The good news: Arrow's CI is already in good shape
>> > (path filters, concurrency cancelling, the "CI: Extra" labels,
>> > RunsOn for CUDA). Even so, I found some easy wins, and I'd love to
>> > hear what you think.
>> >
>> > I was a listener at yesterday's board meeting and i understand
>> > that Arrow has some CI "difficulties" and is looking for ways to
>> > deal with and JBO was generally offering his overlook of that
>> > (hence the CC:) but then there was a lot of buzz about the
>> > #project-workflows-optimisations channel of ASF - and I just added
>> > new "workflow-optimization" skill in Magpie that is
>> > producing very good advice, and you can use it to implement changes.
>> > those optimizations quickly.
>> >
>> > Last week we used roughly 110-125k GitHub-hosted runner minutes.
>> > Package Linux alone is about 40% of that.
>>
>> Fedora does offer CI builds for packaging:
>> https://packit.dev/
>>
>> Probably CentOS and other RPM builds could be moved to that.
>>
>
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Andrew Lamb <an...@gmail.com>.
(also adding dev@datafusion as they are affected as well)
Hi Jarek,
For others who are looking for more context related to this email, there is
an ASF slack thread[1] and issue[2] with more background and data.
> And we are explaining that for weeks, yet, for some reason we get nudges
from Andrew every day - even if he knows very well we are working on it.
The reason I am nudging is that for the last several days, contributors in
arrow-rs and datafusion are waiting 3-7 hours for CI jobs to start. I have
heard frustration from many community members. This seems to have
gotten much worse in the last week. While waiting for jobs to start, I
started investigating what projects were using the runners that our jobs
were waiting behind (the script I use is attached to issue [2]) which lead
me to Airflow. I am sorry if it seems I am singling that project out but it
seems to be using significantly more than the next largest project I can
find.
> (Airflow and Arrow) are two biggest users of the CI in ASF.
Can you please share the data that you used to come to this conclusion? My
numbers[3] suggest over the last 24 hours Airflow has used roughly 10x more
shared runner time (106 runner-day vs 16 runner-days) than
arrow/arrow-rs/arrow-go.
> Andrew, are you actually working with other Arrow team members on
applying those improvements I proposed (or other?). It seems a bit one sided
when you have all proactiveness from our side, yet we do not hear what you
are doing to help the ASF.
I am not sure what you proposed as I don't use or follow Airflow; I can
tell you that both arrow-rs and DataFusion communities have been busy
trying to reduce our shared runner usage, I have listed a few examples
below but you can find a bunch more if you search for recent merge PRs in
those repositories
Andrew
[1]: https://the-asf.slack.com/archives/CBX4TSBQ8/p1792520326623569
[2]: https://github.com/apache/datafusion/issues/25088
[3]:
https://github.com/apache/datafusion/issues/25088#issuecomment-6369916889
DataFusion: PRs to reduce shared runner load
- https://github.com/apache/datafusion/pull/25591 (Oct 20) Runs the Rust
workflow for fork PRs on the contributor's fork.
- https://github.com/apache/datafusion/pull/26574 (Oct 21) Moves the
circular dependency check onto the fork
- https://github.com/apache/datafusion/pull/26571 (Oct 21) Moves about a
dozen small jobs to ubuntu-slim.
Arrow (Rust): PRs to reduce shared runner load
- https://github.com/apache/arrow-rs/pull/11788: "Run fork PR test suites
in contributor forks".
- https://github.com/apache/arrow-rs/pull/11774: "ci: move several jobs to
ubuntu-slim runner".
- https://github.com/apache/arrow-rs/pull/11631: "Run Miri on a daily
schedule, require it before releases".
On Fri, Oct 23, 2026 at 12:27 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> Hello here,
>
> I did not want to nudge anyone here, but since Andrew does it. Just
> adding Andrew - who keeps on nudging Airflow in issues - we are working in
> Airflow on implementing **way** more improvements than that - for several
> weeks already - and we are close to conclusion.
> And the only way to deal with queueing is to get big users (Airflow and
> Arrow) are two biggest users of the CI in ASF. In Airflow we are working on
> 80% cutting - via using self-hosted runners and via delegating CI to the
> contributor's public runners.
>
> And we are explaining that for weeks, yet, for some reason we get nudges
> from Andrew every day - even if he knows very well we are working on it.
>
> And - just to be sure - those 30% - 40% savings above are easily
> achievable (way easier than the 80% of Airflow) - and it is also absolutely
> necessary to remove the pressure from the queue.,
>
> Andrew,
>
> Andrew, are you actually working with other Arrow team members on applying
> those improvements I proposed (or other?). It seems a bit one sided when
> you have all proactiveness from our side, yet we do not hear what you are
> doing to help the ASF.
>
> I would really appreciate **some** answer here how are you working with
> others in the Arrow to solve the issues. Are there any plans to improve
> things ?
>
> And yes Andrew - I expect answer from you,
>
> J.
>
> On Thu, Oct 22, 2026 at 8:46 PM Benson Muite <be...@emailplus.org>
> wrote:
>
>> Jarek Potiuk <ja...@potiuk.com> writes:
>>
>> Hi,
>>
>> > Hi everyone,
>> >
>> > I hope you're all doing well! I've been looking at how Arrow uses
>> > GitHub Actions. All ASF projects share the same pool of GitHub
>> > runners, so every minute we save also shortens the queue for
>> > everyone else. The good news: Arrow's CI is already in good shape
>> > (path filters, concurrency cancelling, the "CI: Extra" labels,
>> > RunsOn for CUDA). Even so, I found some easy wins, and I'd love to
>> > hear what you think.
>> >
>> > I was a listener at yesterday's board meeting and i understand
>> > that Arrow has some CI "difficulties" and is looking for ways to
>> > deal with and JBO was generally offering his overlook of that
>> > (hence the CC:) but then there was a lot of buzz about the
>> > #project-workflows-optimisations channel of ASF - and I just added
>> > new "workflow-optimization" skill in Magpie that is
>> > producing very good advice, and you can use it to implement changes.
>> > those optimizations quickly.
>> >
>> > Last week we used roughly 110-125k GitHub-hosted runner minutes.
>> > Package Linux alone is about 40% of that.
>>
>> Fedora does offer CI builds for packaging:
>> https://packit.dev/
>>
>> Probably CentOS and other RPM builds could be moved to that.
>>
>
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Jarek Potiuk <ja...@potiuk.com>.
Hello here,
I did not want to nudge anyone here, but since Andrew does it. Just adding
Andrew - who keeps on nudging Airflow in issues - we are working in Airflow
on implementing **way** more improvements than that - for several weeks
already - and we are close to conclusion.
And the only way to deal with queueing is to get big users (Airflow and
Arrow) are two biggest users of the CI in ASF. In Airflow we are working on
80% cutting - via using self-hosted runners and via delegating CI to the
contributor's public runners.
And we are explaining that for weeks, yet, for some reason we get nudges
from Andrew every day - even if he knows very well we are working on it.
And - just to be sure - those 30% - 40% savings above are easily achievable
(way easier than the 80% of Airflow) - and it is also absolutely necessary
to remove the pressure from the queue.,
Andrew,
Andrew, are you actually working with other Arrow team members on applying
those improvements I proposed (or other?). It seems a bit one sided when
you have all proactiveness from our side, yet we do not hear what you are
doing to help the ASF.
I would really appreciate **some** answer here how are you working with
others in the Arrow to solve the issues. Are there any plans to improve
things ?
And yes Andrew - I expect answer from you,
J.
On Thu, Oct 22, 2026 at 8:46 PM Benson Muite <be...@emailplus.org>
wrote:
> Jarek Potiuk <ja...@potiuk.com> writes:
>
> Hi,
>
> > Hi everyone,
> >
> > I hope you're all doing well! I've been looking at how Arrow uses
> > GitHub Actions. All ASF projects share the same pool of GitHub
> > runners, so every minute we save also shortens the queue for
> > everyone else. The good news: Arrow's CI is already in good shape
> > (path filters, concurrency cancelling, the "CI: Extra" labels,
> > RunsOn for CUDA). Even so, I found some easy wins, and I'd love to
> > hear what you think.
> >
> > I was a listener at yesterday's board meeting and i understand
> > that Arrow has some CI "difficulties" and is looking for ways to
> > deal with and JBO was generally offering his overlook of that
> > (hence the CC:) but then there was a lot of buzz about the
> > #project-workflows-optimisations channel of ASF - and I just added
> > new "workflow-optimization" skill in Magpie that is
> > producing very good advice, and you can use it to implement changes.
> > those optimizations quickly.
> >
> > Last week we used roughly 110-125k GitHub-hosted runner minutes.
> > Package Linux alone is about 40% of that.
>
> Fedora does offer CI builds for packaging:
> https://packit.dev/
>
> Probably CentOS and other RPM builds could be moved to that.
>
Re: [DISCUSS] Ideas to make Arrow's GitHub Actions CI lighter (~30-40% fewer runner minutes)
Posted by Benson Muite <be...@emailplus.org>.
Jarek Potiuk <ja...@potiuk.com> writes:
Hi,
> Hi everyone,
>
> I hope you're all doing well! I've been looking at how Arrow uses
> GitHub Actions. All ASF projects share the same pool of GitHub
> runners, so every minute we save also shortens the queue for
> everyone else. The good news: Arrow's CI is already in good shape
> (path filters, concurrency cancelling, the "CI: Extra" labels,
> RunsOn for CUDA). Even so, I found some easy wins, and I'd love to
> hear what you think.
>
> I was a listener at yesterday's board meeting and i understand
> that Arrow has some CI "difficulties" and is looking for ways to
> deal with and JBO was generally offering his overlook of that
> (hence the CC:) but then there was a lot of buzz about the
> #project-workflows-optimisations channel of ASF - and I just added
> new "workflow-optimization" skill in Magpie that is
> producing very good advice, and you can use it to implement changes.
> those optimizations quickly.
>
> Last week we used roughly 110-125k GitHub-hosted runner minutes.
> Package Linux alone is about 40% of that.
Fedora does offer CI builds for packaging:
https://packit.dev/
Probably CentOS and other RPM builds could be moved to that.