Re: [ANNOIUNCE] Self-hosted runners and "Your PR - Your CI" in tests

[Jarek Potiuk <ja...@potiuk.com>]

Hello dear fellow Airflow maintainers.

Following up on that recent request: to anyone cherry-picking to v3-4-test
and to Rahul—please direct all work related to 3.4.0 from airflow/apache,
not from your forks.

This should be not only faster and have less queueing during the workdays
but also it will free a lot of queue time. We implemented **a number** of
optimisations and shared them here:
https://infra.apache.org/github-actions-optimization.html - including the
latest optimisation:
https://github.com/apache/infrastructure-website/pull/323 - automated
cancelling of workflows that are from PRs merged "forcefully".

Any issues - let us know at #internal-airflow-ci-cd channel in our slack.

Hello dear infra users, Datafusion and Arrow teams,

As you know, Airflow, due to its 3.4.0 release combined with the PR
flood was one of the biggest contributors to the queue - in ASF. This is
why during last two weeks we worked proactively on those optkimisations and
since last weekend was free of nudging and interruptions, Shahar and I were
able to focus on getting it done. It's well advanced and we will gradually
turn the knobs over next few days to get lower and lower usage of shared
resources -at the expense of our self-hosted runners and CI time of our
contributors.

Andrew Lamb -> NOW is the time you start sending - possibly in that thread
- results of your monitoring, also I think Arrow and Datafusion who are
probably the next will also implement some of the optimisations I proposed
- and we will - together get back to "normal" state.

Andrew - please let us know what your monitoring will show.

J.


On Mon, Oct 26, 2026 at 2:35 AM Jarek Potiuk <ja...@potiuk.com> wrote:

> Hello everyone
>
> Thanks to alll the hard work by Shajar -  we have now self-hosted runners
> enabled and **working** (it seems).. We've been implementing and testing it
> over the weekend together with Shahar, and solved some "teetihing" problems.
>
> Right now, self-hosted runners are enabled for:
>
> * canary / scheduled runs from main
> * PRs of maintainer from "airflow" repo
>
> We are still experimenting and fine-tuning. For example, Shahar added big
> runners (128 GB Memory) that should significantly shorten the elapsed time
> for running tests.
>
> We still need to address some issues, such as spot instance eviction, but
> runs-on and some custom CI steps should handle those nicely.
>
> We are also in the advanced testing stage with 'using fork runners' for CI
> - still needs a few fixes, but basic assumptions seem to hold well.
>
> More tests tomorrow.
>
> J.
>