You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/10/20 03:43:54 UTC

[PR] ci: parallelize the merge queue and stop gating CI jobs on lint [iceberg-rust]

kevinjqliu opened a new pull request, #3490:
URL: https://github.com/apache/iceberg-rust/pull/3490

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
   -->
   
   - Closes N/A. Related to #3442, split out of #3221.
   
   ## What changes are included in this PR?
   
   <!--
   Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates.
   -->
   
   Merge queue CI took ~15 min median and 75+ min p90 over the last week, for ~4.5 min of compute. Most of that is waiting for `ubuntu-24.04` runners, which are contended across ASF. This PR cuts how often we pay that wait:
   
   - Build and merge up to 3 merge queue entries at once. With 1, queued PRs wait serially. 3 queued PRs pay 3 x (wait + run). `ALLGREEN` still requires every entry to pass.
   - Drop `needs: lint` from CI jobs, so each run queues once instead of twice. In one run, lint waited 49 min, then every other Linux job waited another 51 min. `ci-required` still needs `lint`.
   
   ## Are these changes tested?
   
   <!--
   Specify what test covers (unit test, integration test, etc.).
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   `actionlint` (no new findings) and `zizmor --offline` (clean).
   
   ## AI Disclosure
   
   <!--
   https://iceberg.apache.org/contribute/#guidelines-for-ai-assisted-contributions
   -->
   
   Prepared with GitHub Copilot. Check that 3 is a reasonable queue width, and that jobs still running after a lint failure is an acceptable trade-off.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] ci: build and merge up to three merge queue entries at once [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #3490:
URL: https://github.com/apache/iceberg-rust/pull/3490#discussion_r4350623771


##########
.github/workflows/ci.yml:
##########
@@ -217,7 +213,6 @@ jobs:
         run: cargo build -p iceberg --no-default-features
 
   tests:
-    needs: lint

Review Comment:
   Good call. I pulled numbers for the last 2 weeks. On PR CI, lint failed in 20 of 357 runs (5.6%), and skipping the dependent jobs saved ~590 runner-min, about 5.6% of PR CI compute. In the merge queue, lint never failed across 79 runs, so there `needs: lint` only adds a second runner wait.
   
   Dropping it only for the merge queue is doable, but `needs` can't depend on the event. We'd have to skip `lint` on `merge_group`, add a second lint job for it, add an `if` to every dependent job, and have `ci-required` accept the skipped one. I reverted the `needs: lint` change to keep this PR to the merge queue setting. Happy to do the split in a follow-up if you think it's worth it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] ci: build and merge up to three merge queue entries at once [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #3490:
URL: https://github.com/apache/iceberg-rust/pull/3490#discussion_r4351133366


##########
.github/workflows/ci.yml:
##########
@@ -217,7 +213,6 @@ jobs:
         run: cargo build -p iceberg --no-default-features
 
   tests:
-    needs: lint

Review Comment:
   yea i put it back. you're right about reducing workload. We technically dont need to block on `lint`, but doing so allow us to block on it immediate instead of running all the jobs in parallel and saturate the already demand runners 
   
   keeping as is, which is like this:
   
   <img width="627" height="515" alt="Screenshot 2026-10-22 at 9 56 41 AM" src="https://github.com/user-attachments/assets/8e7a9883-f4c7-48c2-915b-4a3f736147a1" />
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] ci: parallelize the merge queue and stop gating CI jobs on lint [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #3490:
URL: https://github.com/apache/iceberg-rust/pull/3490#discussion_r4321262943


##########
.github/workflows/ci.yml:
##########
@@ -109,7 +109,6 @@ jobs:
         uses: crate-ci/typos@512fc24f32f44ab01972217aaaf3dc86ec234d53 # v1.50.2
 
   clippy:
-    needs: lint

Review Comment:
   let these parallelize 



##########
.asf.yaml:
##########
@@ -95,8 +95,8 @@ github:
           parameters:
             check_response_timeout_minutes: 360
             grouping_strategy: ALLGREEN
-            max_entries_to_build: 1
-            max_entries_to_merge: 1
+            max_entries_to_build: 3
+            max_entries_to_merge: 3

Review Comment:
   set parallelism on merge queue so we can test prs together



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] ci: parallelize the merge queue and stop gating CI jobs on lint [iceberg-rust]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3490:
URL: https://github.com/apache/iceberg-rust/pull/3490#discussion_r4321329437


##########
.asf.yaml:
##########
@@ -95,8 +95,8 @@ github:
           parameters:
             check_response_timeout_minutes: 360
             grouping_strategy: ALLGREEN
-            max_entries_to_build: 1
-            max_entries_to_merge: 1
+            max_entries_to_build: 3
+            max_entries_to_merge: 3

Review Comment:
   The choice of `3` here is a policy/magic-number that will likely need future tuning. Add a short inline comment explaining why 3 was selected (e.g., based on observed runner wait times / desired throughput) and what signal would justify adjusting it up/down. This helps future maintainers avoid guesswork when revisiting queue performance.



##########
.asf.yaml:
##########
@@ -95,8 +95,8 @@ github:
           parameters:
             check_response_timeout_minutes: 360
             grouping_strategy: ALLGREEN
-            max_entries_to_build: 1
-            max_entries_to_merge: 1
+            max_entries_to_build: 3
+            max_entries_to_merge: 3
             merge_method: SQUASH

Review Comment:
   `max_entries_to_merge: 3` increases how many entries can land back-to-back without revalidating the base branch between each individual merge (depending on how the ASF merge queue implementation batches merges under `ALLGREEN`). Consider keeping `max_entries_to_build` at 3 for throughput, but leaving `max_entries_to_merge` at 1 to preserve more conservative merge serialization (lower risk of rapid consecutive merges causing follow-on breakage/conflicts). If you keep `max_entries_to_merge: 3`, it would be good to document the expected batching/merge semantics and the operational trade-off being made.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] ci: build and merge up to three merge queue entries at once [iceberg-rust]

Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk merged PR #3490:
URL: https://github.com/apache/iceberg-rust/pull/3490


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] ci: parallelize the merge queue and stop gating CI jobs on lint [iceberg-rust]

Posted by "blackmwk (via GitHub)" <gi...@apache.org>.
blackmwk commented on code in PR #3490:
URL: https://github.com/apache/iceberg-rust/pull/3490#discussion_r4346675792


##########
.github/workflows/ci.yml:
##########
@@ -217,7 +213,6 @@ jobs:
         run: cargo build -p iceberg --no-default-features
 
   tests:
-    needs: lint

Review Comment:
   Could we keep the dependencices on lints? It helps us reduce workload in pull request ci. Or ideally we could remove the dependency on merge queue run, and still add it on pr ci run. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] ci: build and merge up to three merge queue entries at once [iceberg-rust]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #3490:
URL: https://github.com/apache/iceberg-rust/pull/3490#discussion_r4350623771


##########
.github/workflows/ci.yml:
##########
@@ -217,7 +213,6 @@ jobs:
         run: cargo build -p iceberg --no-default-features
 
   tests:
-    needs: lint

Review Comment:
   Good call. I pulled numbers for the last 2 weeks. On PR CI, lint failed in 20 of 357 runs (5.6%), and skipping the dependent jobs saved ~590 runner-min, about 5.6% of PR CI compute. In the merge queue, lint never failed across 79 runs, so there `needs: lint` only adds a second runner wait.
   
   Dropping it only for the merge queue is doable, but `needs` can't depend on the event. We'd have to skip `lint` on `merge_group`, add a second lint job for it, add an `if` to every dependent job, and have `ci-required` accept the skipped one. I reverted the `needs: lint` change to keep this PR to the merge queue setting. Happy to do the split in a follow-up if you think it's worth it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org