You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/10/14 13:48:48 UTC
[I] Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch [datafusion]
alamb opened a new issue, #26292:
URL: https://github.com/apache/datafusion/issues/26292
### Describe the bug
The merge queue regularly removes PRs with "removed this pull request from the merge queue due to no response for status checks" even though **no check failed**.
For example, https://github.com/apache/datafusion/pull/26244 is a 2-line change. It was approved on 2026-10-12 at 18:12 UTC. Since then it has been added to the merge queue **7 times by 4 different committers** and removed **6 times**.
| Added to queue (UTC) | By | Removed (UTC) | Time in queue |
|---|---|---|---|
| 10-12 19:09 | @alamb | 10-12 19:14 | 5 min |
| 10-13 10:05 | @alamb | 10-13 12:51 | 2 h 46 min (timed out) |
| 10-13 14:36 | @mbutrovich | 10-13 18:19 | 3 h 43 min (timed out) |
| 10-13 18:22 | @mbutrovich | 10-13 20:31 | 2 h 09 min |
| 10-14 07:25 | @Jefffrey | 10-14 08:28 | 1 h 03 min (timed out) |
| 10-14 08:50 | @Jefffrey | 10-14 12:40 | 3 h 50 min |
| 10-14 12:45 | @mbutrovich | (still queued) | |
In every case I looked at, checks had already passed and the rest had simply not finished (or started) within the hour. When the PR is added back to the queue, all of the CI runs again from the beginning, and often times out again.
This has been happening on and off for a while (see #23400 from September), but over the last two days (2026-10-13 and 2026-10-14) it affected most PRs that went through the queue.
### Problems this causes
**1. Merging an approved PR takes a long time and several manual steps**
See above. Other recent examples of PRs that timed out in the queue (all eventually passed CI unchanged):
| PR | Queue attempts | Time from first queue to merge | People who had to re-queue it |
|---|---|---|---|
| #26251 | 5 | 19.6 h | 2 |
| #26253 | 4 | 19.6 h | 2 |
| #25494 | 3 | 20.6 h | 2 |
| #26132 | 2 | 17.3 h | 2 |
| #26215 | 2 | 11.4 h | 2 |
| #25891 | 2 | 7.0 h | 1 |
| #26110 | 4 so far | still open (first queued 10-09) | 1 |
| #25710 | 2 so far | still open | 1 |
| #26092, #26096, #26224, #26164, #26010, #24093 | 1-2 so far | each timed out at least once on 10-13 / 10-14 | |
**2. It burns a lot of CI capacity, because the tests do eventually finish but have to run again**
Each timeout throws away CI work that already completed.
For each timeout, 22 jobs (65 to 88 runner-minutes of work) had already passed when the PR was evicted. When the PR is re-queued, those 22 jobs run again along with everything else.
For the 19 timeouts I checked on 10-13 / 10-14 alone, that is roughly 23 runner-hours of passing CI discarded in about 36 hours. We are already a top ASF consumer of GitHub Actions minutes:
- #25148
Even worse is that these timeouts also happen precisely when runners are scarce, so the re-runs add load at the worst possible time and make the next timeout more likely.
### Expected behavior
A PR whose checks are all passing, just slowly, should merge without anyone having to touch it again. CI work that has already passed for a queue entry should not be discarded and redone.
### Additional context
- The queue's status check timeout is currently 60 minutes (the GitHub default); the evictions above all happened 60 to 80 minutes after the queue built the entry.
- Related: #23400 (earlier queue evictions, closed),
- #25088 (CI takes hours to start),
- #25148 (reduce GitHub Actions usage)
- #6880 (original merge queue discussion).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #26292:
URL: https://github.com/apache/datafusion/issues/26292#issuecomment-6173153719
I'm wondering if MQ is shared, because the similar has been reported in `iceberg-rs` repo yesterday
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #26292:
URL: https://github.com/apache/datafusion/issues/26292#issuecomment-6175120080
> 4\. I'm wondering if MQ is shared, because the similar issue has been reported in `iceberg-rs` repo yesterday [apache/iceberg-rust#3425](https://github.com/apache/iceberg-rust/pull/3425)
@comphead I think the issue is that the github runners are shared across apache projects somehow -- so I think we are suffering from long queue wait times as the runner pool is at max capacity
> 3\. https://github.com/apache/datafusion/actions/runs/38528268442/job/115733699502 - we were waiting 87 minutes for this run to start - even though the action itself took just 10 s 🤯
Yeah, @blaginin and even worse is if that then kicks the PR out of the merge queue, we have to re-run all CI again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #26292:
URL: https://github.com/apache/datafusion/issues/26292#issuecomment-6175097237
> I feel we need to reduce MQ throughput for DF, Comet is already 5 max builds
Maybe we could in theory also reduce the number of concurrent builds to reduce the load on the runners 🤔 (aka down to 3 or something). but that would in theory reduce the throughput and not reduce the overall demand)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on issue #26292:
URL: https://github.com/apache/datafusion/issues/26292#issuecomment-6172868249
https://github.com/apache/datafusion/actions/runs/38528268442/job/115733699502 - we were waiting 87 minutes for this run to start - even though the action itself took just 10 🤯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #26292: Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch
URL: https://github.com/apache/datafusion/issues/26292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #26292:
URL: https://github.com/apache/datafusion/issues/26292#issuecomment-6173871656
I feel we need to reduce MQ throughput for DF and Comet
```
max_entries_to_merge: 5
max_entries_to_build: 5
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on issue #26292:
URL: https://github.com/apache/datafusion/issues/26292#issuecomment-6173243911
i think it's not about MQ, but instead of ASF runners being overloaded...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #26292:
URL: https://github.com/apache/datafusion/issues/26292#issuecomment-6297585924
merge queue timeout is merged to fix this one
- https://github.com/apache/datafusion-sqlparser-rs/pull/2638
The overall queue time is not fixed, that is tracked with
- https://github.com/apache/datafusion/issues/25088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org