[DISCUSS] Run CI in a merge queue instead of on every push

[Walter Duque de Estrada via dev <de...@grails.apache.org>]

[DISCUSS] Run CI in a merge queue instead of on every push

Idea: a push to a PR builds nothing. When a PR is ready, "Merge when ready" puts it in a merge queue, which runs the full CI on the merged tree and merges only if it is green. A failed build retries itself once before it counts as failed.

Why: today every push to a PR costs ~40 runner-hours and we average almost 4 pushes per PR. Testing once per merge cuts that by about two thirds and removes the case we hit with #16656, a merge whose tests never ran. Eight ASF projects (datafusion, arrow-rs, seatunnel, ...) already run merge queues from .asf.yaml, so no Infra request is needed.

Trade-off: feedback arrives later. A contributor sees failures when the PR is queued, not when they push, unless they ask for a run with the `run ci` label. Merging also takes longer, since the queue has to wait for a runner.

Questions for the team:
1. Are we fine with no checks at all on PR pushes, or do we keep the cheap ones (code style, build health)?
2. Default branch first, or all release branches at once?

Re: [DISCUSS] Run CI in a merge queue instead of on every push

[Walter Duque de Estrada via dev <de...@grails.apache.org>]

What you says makes sense. Some of the issues I personally have had is because of the size o
of the rework and the shape of the rework. Replacing all the groovy to java to make it more modular
and not lose track of the changes in git grew the size quadratically. They should not be DRAFT because they
wont get looked at; if labellled no-ci and it gets merged the tests dont run.

Send me the draft of what you were doing or a prompt and I can start on it.


Thanks

Walter

> On Oct 22, 2026, at 11:42 AM, James Daugherty via dev <de...@grails.apache.org> wrote:
> 
> Hi Walter,
> 
> I think it's important to run CI on PRs for the default branch and higher
> (i.e. active development). The problem is capacity.  We added an opt out
> mechanism to silence PRs that receive too many frequent updates, but we
> still run tests by default for active branches. If we know we're going to
> iterate on a PR, or continue updating it, I suggest making it a draft or
> adding the "no ci" label to suppress runs until it's ready.  This gives us
> the choice, which a merge queue would not.
> 
> The later feedback is also a major problem; we've tried this before.  When
> we had separate repos, that's effectively what happened.  Someone would
> merge into core, and then the functional tests would later fail or require
> updates.  As someone currently doing the majority of major branch merging,
> I already struggle to keep the branches up-to-date.  I don't want to make
> that process even more difficult.
> 
> Self-hosted builds are the solution to the capacity problem. I've done some
> initial cost analysis but I need to spend more time on it, and I cannot
> currently.  I'm instead focusing on getting the 8.1 & 9.0-M1 builds ready.
> If someone else has time to continue that work, I'm happy to pass it off.
> 
> -James
> 
> On Thu, Oct 22, 2026 at 11:30 AM Walter Duque de Estrada via dev <
> dev@grails.apache.org> wrote:
> 
>> [DISCUSS] Run CI in a merge queue instead of on every push
>> 
>> Idea: a push to a PR builds nothing. When a PR is ready, "Merge when
>> ready" puts it in a merge queue, which runs the full CI on the merged tree
>> and merges only if it is green. A failed build retries itself once before
>> it counts as failed.
>> 
>> Why: today every push to a PR costs ~40 runner-hours and we average almost
>> 4 pushes per PR. Testing once per merge cuts that by about two thirds and
>> removes the case we hit with #16656, a merge whose tests never ran. Eight
>> ASF projects (datafusion, arrow-rs, seatunnel, ...) already run merge
>> queues from .asf.yaml, so no Infra request is needed.
>> 
>> Trade-off: feedback arrives later. A contributor sees failures when the PR
>> is queued, not when they push, unless they ask for a run with the `run ci`
>> label. Merging also takes longer, since the queue has to wait for a runner.
>> 
>> Questions for the team:
>> 1. Are we fine with no checks at all on PR pushes, or do we keep the cheap
>> ones (code style, build health)?
>> 2. Default branch first, or all release branches at once?

Re: [DISCUSS] Run CI in a merge queue instead of on every push

[James Daugherty via dev <de...@grails.apache.org>]

Hi Walter,

I think it's important to run CI on PRs for the default branch and higher
(i.e. active development). The problem is capacity.  We added an opt out
mechanism to silence PRs that receive too many frequent updates, but we
still run tests by default for active branches. If we know we're going to
iterate on a PR, or continue updating it, I suggest making it a draft or
adding the "no ci" label to suppress runs until it's ready.  This gives us
the choice, which a merge queue would not.

The later feedback is also a major problem; we've tried this before.  When
we had separate repos, that's effectively what happened.  Someone would
merge into core, and then the functional tests would later fail or require
updates.  As someone currently doing the majority of major branch merging,
I already struggle to keep the branches up-to-date.  I don't want to make
that process even more difficult.

Self-hosted builds are the solution to the capacity problem. I've done some
initial cost analysis but I need to spend more time on it, and I cannot
currently.  I'm instead focusing on getting the 8.1 & 9.0-M1 builds ready.
If someone else has time to continue that work, I'm happy to pass it off.

-James

On Thu, Oct 22, 2026 at 11:30 AM Walter Duque de Estrada via dev <
dev@grails.apache.org> wrote:

> [DISCUSS] Run CI in a merge queue instead of on every push
>
> Idea: a push to a PR builds nothing. When a PR is ready, "Merge when
> ready" puts it in a merge queue, which runs the full CI on the merged tree
> and merges only if it is green. A failed build retries itself once before
> it counts as failed.
>
> Why: today every push to a PR costs ~40 runner-hours and we average almost
> 4 pushes per PR. Testing once per merge cuts that by about two thirds and
> removes the case we hit with #16656, a merge whose tests never ran. Eight
> ASF projects (datafusion, arrow-rs, seatunnel, ...) already run merge
> queues from .asf.yaml, so no Infra request is needed.
>
> Trade-off: feedback arrives later. A contributor sees failures when the PR
> is queued, not when they push, unless they ask for a run with the `run ci`
> label. Merging also takes longer, since the queue has to wait for a runner.
>
> Questions for the team:
> 1. Are we fine with no checks at all on PR pushes, or do we keep the cheap
> ones (code style, build health)?
> 2. Default branch first, or all release branches at once?