You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/29 13:53:36 UTC
[I] ci: raise the merge queue's build concurrency from 2 to 4 [datafusion-comet]
andygrove opened a new issue, #6386:
URL: https://github.com/apache/datafusion-comet/issues/6386
### What is the problem the feature request solves?
The merge queue builds at most two entries at once (`max_entries_to_build: 2` in `.asf.yaml`). Between 2026-09-11 and 2026-09-29 a full queue build took a median of 92 minutes (p90 117), so even a queue that never runs dry tops out at roughly 30 merges a day. Over the last week PRs were approved at about 20 a day and merged at about 13.6 a day, so the backlog keeps growing, and when a batch of PRs is queued together the last one lands 10 to 12 hours later.
A simulation that replays the observed build durations and the observed ~12% per-entry failure rate gives:
| `max_entries_to_build` | Merges/day, queue never empty | Build time lost to rebuilds after a failure |
| --- | --- | --- |
| 2 | ~30 | ~11% |
| 3 | ~44 | ~14% |
| 4 | ~58 | ~16% |
| 6 | ~82 | ~21% |
### Describe the potential solution
Set `max_entries_to_build: 4`. The comment on the setting already anticipates this ("Raise it if the queue backs up"). The runner cost of a merge does not change. What grows is the waste when an entry fails, because the entries behind it that were already building have to start again.
While there, correct the comment on the `merge_group` branch of the `Detect changes` step in `ci.yml`. It says `merge_group.base_sha` covers every entry batched into the group, but `base_sha` is the queue commit of the entry ahead, not `main`. For example, `gh-readonly-queue/main/pr-6357-1e08e6c9` was built on `1e08e6c9`, which was #6300's queue commit. Each entry's build therefore runs only the suites that its own changed files select. That is sound under `ALLGREEN`, and it is why switching to `HEADGREEN` would not be: a green entry could carry an earlier entry whose suites never ran.
### Additional context
Runner budget: an estimated ~190k runner-minutes a week today (PR runs ~96k, queue ~68k, label runs ~10k, nightly ~9k) against the ASF cap of 250k a week. Concurrency does not change the cost per merge, but merging more PRs a week does.
Related: #5969 (cancel a queue run on its first failure, which frees the slot sooner).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ci: raise the merge queue's build concurrency from 2 to 4 [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #6386: ci: raise the merge queue's build concurrency from 2 to 4
URL: https://github.com/apache/datafusion-comet/issues/6386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org