You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/31 12:17:24 UTC
[PR] fix: use single spill file per partition in sort-based shuffle [datafusion-ballista]
andygrove opened a new pull request, #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431
## Summary
- Keeps one `StreamWriter` open per output partition in `SpillManager`, appending across multiple spill calls instead of creating a new file each time
- Reduces file descriptor usage and eliminates redundant IPC header/footer overhead from multiple files
- Streams spill data back batch-by-batch via `open_spill_reader()` instead of loading all spilled batches into memory at once
## Test plan
- [x] Existing sort shuffle unit tests pass (`cargo test -p ballista-core sort_shuffle` — 15 tests)
- [x] `cargo clippy` and `cargo fmt` clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3830999849
would something like:
```
sqlbenchmark tpch --iterations 3 --scale-factor 10 -c ballista.shuffle.sort_based.enabled=true -c ballista.shuffle.force_remote_read=true -c datafusion.execution.target_partitions=16 -c ...
```
make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "sqlbenchmark (via GitHub)" <gi...@apache.org>.
sqlbenchmark commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3830543603
## @sqlbenchmark usage
This bot is whitelisted for DataFusion committers.
### Commands
```
@sqlbenchmark tpch [--iterations N] [--scale-factor N] [--baseline BRANCH] [--baseline-pr N]
@sqlbenchmark criterion [--bench NAME] [--filter PATTERN] [--baseline BRANCH] [--baseline-pr N]
@sqlbenchmark help
```
### Examples
```
@sqlbenchmark tpch
@sqlbenchmark tpch --iterations 5 --scale-factor 10
@sqlbenchmark tpch --baseline my-branch
@sqlbenchmark criterion
@sqlbenchmark criterion --bench sql_planner
@sqlbenchmark criterion --baseline my-branch
@sqlbenchmark criterion --bench sort_shuffle --baseline-pr 1430
```
---
*Automated by dfbench*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3830995006
> I would say last benchmark is not correct as spill shuffle is disabled by default.
>
> @andygrove is there a chance we can pass (ballista) configuration values as part of [sqlbenchmark](https://github.com/sqlbenchmark) command?
@milenkovicm I can do that. Could you give me an example of the syntax
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3829490947
@sqlbenchmark criterion --bench sort_shuffle --baseline-pr 1434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3831621059
> would something like:
>
> ```
> sqlbenchmark tpch --iterations 3 --scale-factor 10 -c ballista.shuffle.sort_based.enabled=true -c ballista.shuffle.force_remote_read=true -c datafusion.execution.target_partitions=16 -c ...
> ```
>
> make sense?
I'll need to create a PR to update the benchmark to support this. I am working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#discussion_r2750691854
##########
ballista/core/src/execution_plans/sort_shuffle/spill.rs:
##########
@@ -35,16 +35,17 @@ use std::path::PathBuf;
/// Manages spill files for sort-based shuffle.
///
/// When partition buffers exceed memory limits, they are spilled to disk
-/// as Arrow IPC files. During finalization, these spill files are read
-/// back and merged into the consolidated output file.
-#[derive(Debug)]
+/// as Arrow IPC files. Each output partition has at most one spill file
+/// that is appended to across multiple spill calls. During finalization,
+/// these spill files are read back and merged into the consolidated
+/// output file.
pub struct SpillManager {
/// Base directory for spill files
spill_dir: PathBuf,
- /// Spill files per output partition: partition_id -> Vec<spill_file_path>
- spill_files: HashMap<usize, Vec<PathBuf>>,
- /// Counter for generating unique spill file names
- spill_counter: usize,
+ /// Spill file path per output partition: partition_id -> spill_file_path
Review Comment:
So worst case scenario will be similar to current, non sort spill implementation, having the same number of open files as output partitions?
##########
ballista/core/src/config.rs:
##########
@@ -69,6 +69,9 @@ pub const BALLISTA_SHUFFLE_SORT_BASED_MEMORY_LIMIT: &str =
/// Configuration key for sort shuffle spill threshold (0.0-1.0).
pub const BALLISTA_SHUFFLE_SORT_BASED_SPILL_THRESHOLD: &str =
"ballista.shuffle.sort_based.spill_threshold";
+/// Configuration key for sort shuffle target batch size in rows.
+pub const BALLISTA_SHUFFLE_SORT_BASED_BATCH_SIZE: &str =
+ "ballista.shuffle.sort_based.batch_size";
Review Comment:
Is there any specific reason we don't use data fusions batch size configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "sqlbenchmark (via GitHub)" <gi...@apache.org>.
sqlbenchmark commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3830604985
## Ballista TPC-H Benchmark Results
**PR:** #1431 - fix: add missing batch_size argument to benchmark
**PR Commit:** `89699c71`
**Base Commit:** `4fe9e1d6` (main)
**Scale Factor:** SF10
**Iterations:** 3
### Query Comparison
| Query | main (ms) | PR (ms) | Change |
|-------|-----------|---------|--------|
| Q1 | 2542.20 | 2537.50 | :white_circle: -0.2% |
| Q2 | 1912.10 | 2001.40 | :white_circle: +4.7% |
| Q3 | 2149.10 | 2068.90 | :white_circle: -3.7% |
| Q4 | 1289.40 | 1255.30 | :white_circle: -2.6% |
| Q5 | 4005.80 | 4167.00 | :white_circle: +4.0% |
| Q6 | 917.30 | 908.70 | :white_circle: -0.9% |
| Q7 | 5531.70 | 5464.30 | :white_circle: -1.2% |
| Q8 | 5192.70 | 5182.30 | :white_circle: -0.2% |
| Q9 | 6437.80 | 6355.50 | :white_circle: -1.3% |
| Q10 | 2434.30 | 2467.80 | :white_circle: +1.4% |
| Q11 | 1457.30 | 1493.80 | :white_circle: +2.5% |
| Q12 | 1774.50 | 1766.60 | :white_circle: -0.4% |
| Q13 | 2181.80 | 2195.90 | :white_circle: +0.6% |
| Q14 | 1119.60 | 1113.70 | :white_circle: -0.5% |
| Q15 | 1541.90 | 1506.80 | :white_circle: -2.3% |
| Q16 | 1360.00 | 1339.40 | :white_circle: -1.5% |
| Q17 | 4215.20 | 4328.40 | :white_circle: +2.7% |
| Q18 | 6571.30 | 6584.20 | :white_circle: +0.2% |
| Q19 | 1857.00 | 1803.10 | :white_circle: -2.9% |
| Q20 | 1702.60 | 1743.20 | :white_circle: +2.4% |
| Q21 | 6186.70 | 6260.90 | :white_circle: +1.2% |
| Q22 | 1078.90 | 1065.70 | :white_circle: -1.2% |
**Total:** Main=63459.20ms, PR=63610.40ms (+0.2%)
---
*Automated benchmark run by dfbench*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3830575154
@sqlbenchmark tpch --iterations 3 --scale-factor 10
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "sqlbenchmark (via GitHub)" <gi...@apache.org>.
sqlbenchmark commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3829500007
## Ballista Criterion Benchmark Results
**PR:** #1431 - fix: add missing batch_size argument to benchmark
**PR Commit:** `89699c7`
**Base Commit:** `4fe9e1d` (main)
**Bench:** sort_shuffle
### Main branch
```
.9.34+deprecated
Downloaded serde_derive_internals v0.29.1
Downloaded schemars_derive v0.8.22
Downloaded rustls-pki-types v1.14.0
Downloaded prettyplease v0.2.37
Downloaded datafusion-physical-expr-common v52.1.0
Downloaded datafusion-functions-nested v52.1.0
Downloaded rustls-pemfile v2.2.0
Downloaded pin-project v1.1.10
Downloaded form_urlencoded v1.2.2
Downloaded datafusion-proto-common v52.1.0
Downloaded ring v0.17.14
Downloaded zerofrom v0.1.6
Downloaded yoke-derive v0.8.1
Downloaded want v0.3.1
Downloaded generic-array v0.14.7
Downloaded foldhash v0.1.5
Downloaded futures-timer v3.0.3
Downloaded displaydoc v0.2.5
Downloaded pbjson-build v0.8.0
Downloaded filetime v0.2.27
Downloaded humantime v2.3.0
Downloaded lazy_static v1.5.0
Downloaded fs_extra v1.3.0
Downloaded aws-lc-sys v0.37.0
Downloaded nu-ansi-term v0.50.3
Downloaded nibble_vec v0.1.0
Downloaded ipnet v2.11.0
Downloaded hyper-timeout v0.5.2
Downloaded http-body-util v0.1.3
Downloaded rustls-native-certs v0.8.3
Downloaded parse-display-derive v0.9.1
Downloaded time-macros v0.2.26
Downloaded synstructure v0.13.2
Downloaded tinystr v0.8.2
Downloaded tracing-appender v0.2.4
Downloaded prost-derive v0.14.3
Downloaded hmac v0.12.1
Downloaded httparse v1.10.1
Downloaded hyper-rustls v0.27.7
Downloaded liblzma v0.4.5
Downloaded matchit v0.8.4
Downloaded ferroid v0.8.9
Downloaded strum_macros v0.27.2
Downloaded toml_parser v1.0.6+spec-1.1.0
Downloaded tempfile v3.24.0
Downloaded pbjson-types v0.8.0
Downloaded num-rational v0.4.2
Downloaded litemap v0.8.1
Downloaded icu_provider v2.1.1
Downloaded icu_properties v2.1.2
Downloaded indexmap v1.9.3
Downloaded rstest v0.26.1
Downloaded prost-build v0.14.3
Downloaded libbz2-rs-sys v0.2.2
Downloaded icu_normalizer_data v2.1.1
Downloaded schemars v0.8.22
Downloaded icu_normalizer v2.1.1
Downloaded icu_locale_core v2.1.1
Downloaded miniz_oxide v0.8.9
Downloaded serde_derive v1.0.228
Downloaded substrait v0.62.2
Downloaded icu_collections v2.1.1
Downloaded hyper-util v0.1.19
Downloaded schemars v1.2.0
Downloaded http v1.4.0
Downloaded http v0.2.12
Downloaded hashbrown v0.12.3
Downloaded idna v1.1.0
Downloaded testcontainers v0.26.3
Downloaded itertools v0.14.0
Downloaded iri-string v0.7.10
Downloaded hashbrown v0.15.5
Downloaded hashbrown v0.14.5
Downloaded hyper v1.8.1
Downloaded libm v0.2.16
Downloaded datafusion-substrait v52.1.0
Downloaded h2 v0.4.13
Downloaded icu_properties_data v2.1.2
Downloaded datafusion-sql v52.1.0
Downloaded quick-xml v0.38.4
Downloaded time v0.3.46
Downloaded object_store v0.12.5
Downloaded webpki-roots v1.0.5
Downloaded object v0.32.2
Downloaded nix v0.30.1
Downloaded libmimalloc-sys v0.1.44
Downloaded tracing v0.1.44
Downloaded liblzma-sys v0.4.5
Downloaded linux-raw-sys v0.11.0
error: no bench target named `sort_shuffle` in default-run packages
```
### PR branch
```
Compiling env_logger v0.11.8
Compiling crossbeam-deque v0.8.6
Compiling structopt v0.3.26
Compiling ballista v52.0.0 (/app/ballista/ballista/client)
Compiling ciborium-io v0.2.2
Compiling plotters-backend v0.3.7
Compiling plotters-svg v0.3.7
Compiling ciborium-ll v0.2.2
Compiling itertools v0.10.5
Compiling cast v0.3.0
Compiling criterion-plot v0.5.0
Compiling rayon v1.11.0
Compiling ciborium v0.2.2
Compiling plotters v0.3.7
Compiling tinytemplate v1.2.1
Compiling is-terminal v0.4.17
Compiling oorandom v11.1.5
Compiling anes v0.1.6
Compiling criterion v0.5.1
Compiling ballista-benchmarks v52.0.0 (/app/ballista/benchmarks)
Finished `bench` profile [optimized] target(s) in 6m 05s
Running benches/sort_shuffle.rs (target/release/deps/sort_shuffle-b1db7a23d64cf131)
Gnuplot not found, using plotters backend
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions: Warming up for 3.0000 s
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions: Collecting 10 samples in estimated 8.3379 s (110 iterations)
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions: Analyzing
sort_shuffle_no_spill/10_batches_200_partitions
time: [74.878 ms 75.035 ms 75.212 ms]
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions: Warming up for 3.0000 s
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions: Collecting 10 samples in estimated 6.7437 s (20 iterations)
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions: Analyzing
sort_shuffle_no_spill/50_batches_200_partitions
time: [334.08 ms 335.67 ms 337.27 ms]
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 14.2s.
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit: Collecting 10 samples in estimated 14.171 s (10 iterations)
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit: Analyzing
sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit
time: [1.3970 s 1.4203 s 1.4435 s]
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 19.1s.
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit: Collecting 10 samples in estimated 19.112 s (10 iterations)
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit: Analyzing
sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit
time: [1.9959 s 2.0315 s 2.0692 s]
```
---
*Automated benchmark run by dfbench*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3829506881
@sqlbenchmark criterion --bench sort_shuffle --baseline-pr 1434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#discussion_r2749681236
##########
ballista/core/src/execution_plans/sort_shuffle/buffer.rs:
##########
@@ -110,6 +111,77 @@ impl PartitionBuffer {
pub fn take_batches(&mut self) -> Vec<RecordBatch> {
std::mem::take(&mut self.batches)
}
+
+ /// Drains batches from the buffer, coalescing small batches into
+ /// larger ones up to `target_batch_size` rows each.
+ pub fn drain_coalesced(&mut self, target_batch_size: usize) -> Vec<RecordBatch> {
+ self.memory_used = 0;
+ self.num_rows = 0;
+ let batches = std::mem::take(&mut self.batches);
+ coalesce_batches(batches, &self.schema, target_batch_size)
+ }
+
+ /// Takes all batches, coalescing small batches into larger ones
+ /// up to `target_batch_size` rows each.
+ pub fn take_batches_coalesced(
+ &mut self,
+ target_batch_size: usize,
+ ) -> Vec<RecordBatch> {
+ let batches = std::mem::take(&mut self.batches);
+ coalesce_batches(batches, &self.schema, target_batch_size)
+ }
+}
+
+/// Coalesces small batches into larger ones up to `target_batch_size`
+/// rows each using `concat_batches`.
+fn coalesce_batches(
+ batches: Vec<RecordBatch>,
+ schema: &SchemaRef,
+ target_batch_size: usize,
+) -> Vec<RecordBatch> {
+ if batches.len() <= 1 {
+ return batches;
+ }
+
+ let mut result = Vec::new();
+ let mut pending: Vec<RecordBatch> = Vec::new();
+ let mut pending_rows: usize = 0;
+
+ for batch in batches {
+ let rows = batch.num_rows();
+ if rows >= target_batch_size {
+ // Flush pending small batches first
+ if !pending.is_empty() {
+ if let Ok(merged) = concat_batches(schema, pending.iter()) {
+ result.push(merged);
+ }
+ pending.clear();
+ pending_rows = 0;
+ }
+ result.push(batch);
+ } else {
+ pending_rows += rows;
+ pending.push(batch);
+ if pending_rows >= target_batch_size {
+ if let Ok(merged) = concat_batches(schema, pending.iter()) {
Review Comment:
Thanks! Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3828786671
@sqlbenchmark criterion --bench sort_shuffle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3828911163
@sqlbenchmark criterion --bench sort_shuffle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#discussion_r2749531369
##########
ballista/core/src/execution_plans/sort_shuffle/buffer.rs:
##########
@@ -110,6 +111,77 @@ impl PartitionBuffer {
pub fn take_batches(&mut self) -> Vec<RecordBatch> {
std::mem::take(&mut self.batches)
}
+
+ /// Drains batches from the buffer, coalescing small batches into
+ /// larger ones up to `target_batch_size` rows each.
+ pub fn drain_coalesced(&mut self, target_batch_size: usize) -> Vec<RecordBatch> {
+ self.memory_used = 0;
+ self.num_rows = 0;
+ let batches = std::mem::take(&mut self.batches);
+ coalesce_batches(batches, &self.schema, target_batch_size)
+ }
+
+ /// Takes all batches, coalescing small batches into larger ones
+ /// up to `target_batch_size` rows each.
+ pub fn take_batches_coalesced(
+ &mut self,
+ target_batch_size: usize,
+ ) -> Vec<RecordBatch> {
+ let batches = std::mem::take(&mut self.batches);
+ coalesce_batches(batches, &self.schema, target_batch_size)
+ }
+}
+
+/// Coalesces small batches into larger ones up to `target_batch_size`
+/// rows each using `concat_batches`.
+fn coalesce_batches(
+ batches: Vec<RecordBatch>,
+ schema: &SchemaRef,
+ target_batch_size: usize,
+) -> Vec<RecordBatch> {
+ if batches.len() <= 1 {
+ return batches;
+ }
+
+ let mut result = Vec::new();
+ let mut pending: Vec<RecordBatch> = Vec::new();
+ let mut pending_rows: usize = 0;
+
+ for batch in batches {
+ let rows = batch.num_rows();
+ if rows >= target_batch_size {
+ // Flush pending small batches first
+ if !pending.is_empty() {
+ if let Ok(merged) = concat_batches(schema, pending.iter()) {
+ result.push(merged);
+ }
+ pending.clear();
+ pending_rows = 0;
+ }
+ result.push(batch);
+ } else {
+ pending_rows += rows;
+ pending.push(batch);
+ if pending_rows >= target_batch_size {
+ if let Ok(merged) = concat_batches(schema, pending.iter()) {
Review Comment:
Can this use `LimitedBatchCoalescer`? This uses the `coalesce` api rather than `concat` which is a bit more efficient/precise and future proof.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3830540999
@sqlbenchmark run tpch -s 10 -i 3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "sqlbenchmark (via GitHub)" <gi...@apache.org>.
sqlbenchmark commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3829532619
## Ballista Criterion Benchmark Results
**PR:** #1431 - fix: add missing batch_size argument to benchmark
**PR Commit:** `89699c71`
**Base Commit:** `9f786fc3` (PR #1434)
**Bench:** sort_shuffle
### Baseline (PR #1434)
```
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2026-01-22, rust version 1.93.0 (254b59607 2026-01-19)
info: downloading component 'cargo'
info: downloading component 'rust-std'
info: downloading component 'rustc'
info: installing component 'cargo'
info: installing component 'rust-std'
info: installing component 'rustc'
Gnuplot not found, using plotters backend
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions: Warming up for 3.0000 s
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions: Collecting 10 samples in estimated 6.1474 s (30 iterations)
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions: Analyzing
sort_shuffle_no_spill/10_batches_200_partitions
time: [206.72 ms 207.66 ms 208.58 ms]
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 21.0s.
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions: Collecting 10 samples in estimated 20.977 s (10 iterations)
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions: Analyzing
sort_shuffle_no_spill/50_batches_200_partitions
time: [2.0811 s 2.1597 s 2.2314 s]
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 22.0s.
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit: Collecting 10 samples in estimated 21.968 s (10 iterations)
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit: Analyzing
sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit
time: [2.1551 s 2.1802 s 2.2056 s]
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 24.3s.
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit: Collecting 10 samples in estimated 24.274 s (10 iterations)
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit: Analyzing
sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit
time: [2.3612 s 2.3815 s 2.4034 s]
```
### PR branch
```
Gnuplot not found, using plotters backend
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions: Warming up for 3.0000 s
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions: Collecting 10 samples in estimated 8.4349 s (110 iterations)
Benchmarking sort_shuffle_no_spill/10_batches_200_partitions: Analyzing
sort_shuffle_no_spill/10_batches_200_partitions
time: [73.966 ms 74.378 ms 75.090 ms]
change: [-64.171% -63.726% -63.280%] (p = 0.00 < 0.05)
Performance has improved.
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions: Warming up for 3.0000 s
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions: Collecting 10 samples in estimated 6.9208 s (20 iterations)
Benchmarking sort_shuffle_no_spill/50_batches_200_partitions: Analyzing
sort_shuffle_no_spill/50_batches_200_partitions
time: [346.24 ms 348.48 ms 350.91 ms]
change: [-84.396% -83.864% -83.245%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 10 measurements (40.00%)
2 (20.00%) low mild
1 (10.00%) high mild
1 (10.00%) high severe
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 13.9s.
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit: Collecting 10 samples in estimated 13.898 s (10 iterations)
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit: Analyzing
sort_shuffle_with_spill/50_batches_200_partitions_8mb_limit
time: [1.3633 s 1.3760 s 1.3915 s]
change: [-37.809% -36.887% -35.916%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 18.2s.
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit: Collecting 10 samples in estimated 18.157 s (10 iterations)
Benchmarking sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit: Analyzing
sort_shuffle_with_spill/50_batches_200_partitions_2mb_limit
time: [1.8903 s 1.9251 s 1.9608 s]
change: [-20.922% -19.164% -17.606%] (p = 0.00 < 0.05)
Performance has improved.
```
---
*Automated benchmark run by dfbench*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3829667811
@milenkovicm @mattcuento see previous comments for benchmarks. This PR speeds things up nicely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3830816422
I would say last benchmark is not correct as spill shuffle is disabled by default.
@andygrove is there a chance we can pass (ballista) configuration values as part of [sqlbenchmark](https://github.com/sqlbenchmark) command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Improve sort-based shuffle: single spill file per partition and batch coalescing [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1431:
URL: https://github.com/apache/datafusion-ballista/pull/1431#issuecomment-3828840421
@sqlbenchmark criterion --bench sort_shuffle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org