You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/22 02:25:15 UTC
[PR] fix: rebuild the shuffle writer per iteration in end-to-end benches [datafusion-comet]
andygrove opened a new pull request, #6104:
URL: https://github.com/apache/datafusion-comet/pull/6104
## Which issue does this PR close?
No issue filed. Reported in https://github.com/apache/datafusion-comet/pull/6095#issuecomment-5769660944.
## Rationale for this change
Every end-to-end bench in `native/shuffle/benches/shuffle_writer.rs` panics during criterion warmup:
```
shuffle write error: partition offsets were already published
```
Each bench builds one `ShuffleWriterExec` outside `b.iter()` and executes it on every iteration, but the writer publishes its partition offsets through `PartitionOffsets`, a `OnceLock` that errors on a second `set`. CI does not catch this because `pr_benchmark_check.yml` only runs `cargo check --benches`.
## What changes are included in this PR?
- A `bench_end_to_end` helper that uses `b.iter_batched` to build a fresh exec for each iteration. Construction counts as setup and is not timed.
- The four end-to-end bench loops now call that helper instead of repeating the same execute block.
## How are these changes tested?
Ran `cargo bench -p datafusion-comet-shuffle --bench shuffle_writer -- --quick "end to end"`. All 12 end-to-end benches now finish (compression variants, hash/range/single partitioning, and 200/2000/8000 partitions). Before this change, all of them panicked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: rebuild the shuffle writer per iteration in end-to-end benches [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6104:
URL: https://github.com/apache/datafusion-comet/pull/6104#issuecomment-5777811147
Folded this into #6095 instead, as the last commit on that branch. That PR cannot be benchmarked
at all until this lands, so splitting them just means measuring #6095 against a branch that has to
carry this anyway. The change is unchanged from what was here.
Closing in favour of #6095.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: rebuild the shuffle writer per iteration in end-to-end benches [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #6104: fix: rebuild the shuffle writer per iteration in end-to-end benches
URL: https://github.com/apache/datafusion-comet/pull/6104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org