You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "NoahKusaba (via GitHub)" <gi...@apache.org> on 2026/10/01 19:12:30 UTC

[PR] perf(shuffle-writer): coalesce small batches before writing passthrough shuffle files [datafusion-ballista]

NoahKusaba opened a new pull request, #2518:
URL: https://github.com/apache/datafusion-ballista/pull/2518

   ## Which issue does this PR close?
   
   - Closes #2517.
   
   ## Rationale for this change
   
   The passthrough shuffle writer (`ShuffleWriterExec`) writes every batch it receives as its own Arrow IPC message, with its own compression frame. After a selective filter or a `RepartitionExec`, these batches can be a few hundred rows, which makes shuffle files bigger, adds a compression frame per tiny batch, and gives downstream readers more batches to decode.
   
   ## What changes are included in this PR?
   
   - `utils::write_stream_to_disk` runs batches through arrow's `BatchCoalescer` before `writer.write`, targeting the session `batch_size` (`datafusion.execution.batch_size`, default 8192). This happens inside the existing `spawn_blocking` writer task, so the tokio workers do no extra work.
   - The coalescer is built with `with_biggest_coalesce_batch_size(Some(batch_size / 2))`, so a batch of more than half the target size is written as-is when nothing is buffered. DataFusion uses the same setting in `LimitedBatchCoalescer::new` ([`coalesce/mod.rs#L63-L64`](https://github.com/apache/datafusion/blob/7d3835c71f30cbd3c3ae4041732267f1f453097a/datafusion/physical-plan/src/coalesce/mod.rs#L63-L64)), which `CoalesceBatchesExec` uses ([`coalesce_batches.rs#L230`](https://github.com/apache/datafusion/blob/7d3835c71f30cbd3c3ae4041732267f1f453097a/datafusion/physical-plan/src/coalesce_batches.rs#L230)). Links are to DataFusion `55.1.0`.
   - `PartitionStats.num_batches` now counts the batches actually written to the file, not the batches received.
   - The `sort_shuffle` module doc is corrected: output is one `data.arrow` + `data.arrow.index` pair per task (not per input partition), and every hash-repartitioning stage goes through it.
   
   Rows can now be held in the coalescer, up to `batch_size` rows per output partition, until the stream ends. The bounded channel feeding the writer already held batches the same way.
   
   ## Are there any user-facing changes?
   
   No config changes. Passthrough shuffle files contain fewer, larger batches, and the reported `num_batches` drops to match.
   
   **API change:** the public `ballista_core::utils::write_stream_to_disk` takes a new `batch_size: usize` argument.
   
   ## What is the testing strategy for this PR?
   
   - New `utils::tests::write_stream_to_disk_coalesces_small_batches`: writes 10 batches of 3 rows with `batch_size = 8`, reads the file back and checks for exactly `[0..8, 8..16, 16..24, 24..30]`, plus the stats (30 rows, 4 batches, bytes equal to the file size).
   - New `utils::tests::write_stream_to_disk_passes_large_batches_through`: two 20-row batches with `batch_size = 8` come back unchanged as 2 batches.
   - Updated `shuffle_reader::tests::test_read_local_shuffle`: it previously expected the writer's two 3-row input batches back separately, and now expects the single coalesced 6-row batch.
   - `cargo test` passes for `ballista-core`, `ballista-executor` and `ballista-scheduler`; `cargo clippy --all-targets -D warnings` and `cargo fmt --check` are clean.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf(shuffle-writer): coalesce small batches before writing passthrough shuffle files [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #2518:
URL: https://github.com/apache/datafusion-ballista/pull/2518#discussion_r4160693723


##########
ballista/core/src/utils.rs:
##########
@@ -220,25 +227,36 @@ pub async fn write_stream_to_disk(
 
         let mut writer =
             StreamWriter::try_new_with_options(file, schema.as_ref(), options)?;
+        let mut coalescer = BatchCoalescer::new(schema, batch_size)
+            .with_biggest_coalesce_batch_size(Some(batch_size / 2));
+        let mut num_batches = 0;
 
         while let Some(batch) = rx.blocking_recv() {
             let timer = write_metric.timer();
-            writer.write(&batch)?;
+            coalescer.push_batch(batch)?;

Review Comment:
   [P2] This call can exceed the documented `batch_size`-row per-partition bound. In arrow-select 59.3, if a small partial batch is buffered and the next batch is larger than `batch_size / 2`, the large-batch bypass is not taken while `buffered_rows <= limit`; `push_batch` instead splits the entire input and queues every completed batch internally before returning. The loop below cannot drain those completions until then, so a small-then-large sequence can duplicate/materialize essentially the whole large input at once. The current large-batch test only exercises the empty-coalescer bypass. Please add that transition case and flush/drain the partial batch before pushing an oversized input (or use an API that drains incrementally).



##########
ballista/core/src/utils.rs:
##########
@@ -220,25 +227,36 @@ pub async fn write_stream_to_disk(
 
         let mut writer =
             StreamWriter::try_new_with_options(file, schema.as_ref(), options)?;
+        let mut coalescer = BatchCoalescer::new(schema, batch_size)

Review Comment:
   [P1] Please bound or account for aggregate coalescer memory across output partitions. `ShuffleWriterExec::execute_shuffle_write` invokes this once for every child output partition and starts all K drains concurrently. Range-repartition children expose `execution.target_partitions` here, so one task can have K independent coalescers; this is not limited to the task's active input/vcore count. Each populated coalescer may retain/copy roughly `batch_size` rows in addition to the existing per-partition channels, and none of this is registered with the task memory pool. With the defaults, K=1000 and 1 KiB rows is about 7.8 GiB of new buffering before channel/input overhead. The channel capacity does not cap this state because every output has its own channel and coalescer. Please enforce a shared per-task byte budget / memory reservation and flush partial coalescers under pressure (or otherwise make the total independent of K), and cover a high-K, wide-row case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf(shuffle-writer): coalesce small batches before writing passthrough shuffle files [datafusion-ballista]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2518:
URL: https://github.com/apache/datafusion-ballista/pull/2518#issuecomment-5941434732

   > Requesting changes for two memory-scalability issues in the new coalescing path. The functional behavior and focused tests look good, but aggregate buffering is not bounded for high output-partition counts, and one input sequence can exceed the stated per-partition bound.
   
   Sorry for the trouble, I'll slow down a bit and re-review. :eyes: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf(shuffle-writer): coalesce small batches before writing passthrough shuffle files [datafusion-ballista]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2518:
URL: https://github.com/apache/datafusion-ballista/pull/2518#issuecomment-5938710439

   This was split out of the Ballista performance audit posted in the discord. I singled it out as a PR that would take little to no-time for a reviewer to validate, it also lgtm. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf(shuffle-writer): coalesce small batches before writing passthrough shuffle files [datafusion-ballista]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba closed pull request #2518: perf(shuffle-writer): coalesce small batches before writing passthrough shuffle files
URL: https://github.com/apache/datafusion-ballista/pull/2518


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf(shuffle-writer): coalesce small batches before writing passthrough shuffle files [datafusion-ballista]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2518:
URL: https://github.com/apache/datafusion-ballista/pull/2518#issuecomment-6019412339

   Closing — benchmarking showed this doesn't help any plan Ballista currently produces.
   
   TPC-H SF10 (main @ d6d8bd91 vs this branch, 22 queries): identical results, no time difference beyond noise. Passthrough shuffle files were already well-sized (main: 2,098 batches / 6,631 rows per batch; branch: 2,102 / 6,618), so there was nothing to coalesce.
   
   The motivating case — `UnorderedRangeRepartitionExec` feeding the passthrough writer — isn't planned by any rule today. The range scatter that is planned (parallel windows) writes through `RangeShuffleWriterExec`, and its streaming merge already emits full batches (1M-row window query: 7,936 rows per batch, none under 1,024).
   
   
   I'll be more careful about any other performance PR's in the future, seems like AI isn't ready to 1 shot them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf(shuffle-writer): coalesce small batches before writing passthrough shuffle files [datafusion-ballista]

Posted by "NoahKusaba (via GitHub)" <gi...@apache.org>.
NoahKusaba commented on PR #2518:
URL: https://github.com/apache/datafusion-ballista/pull/2518#issuecomment-5939535515

   Benchmark results for this change. I didn't add the benchmark to the PR; the source is below if anyone wants to reproduce it.
   
   **Setup:** `ShuffleWriterExec` writes 8 partitions × 131,072 rows (1M rows; 8 columns: 4 × Int64, 2 × Float64, 2 × Utf8), default LZ4 codec. Every case slices one batch of distinct rows, so all cases write identical data and only the input batch size changes. "write + read" also reads every file back with `StreamReader`, the way a reducer does. Files are on tmpfs, so disk write-back stays out of the timings. Times are the median of 3 rounds, with `main` (187e21fc) and this branch run alternately.
   
   | Input batches | Measure | main | this PR | Change |
   |---|---|---|---|---|
   | 128 rows | write | 74.1 ms | 41.8 ms | **−44%** |
   | 128 rows | write + read | 132.7 ms | 73.3 ms | **−45%** |
   | 8192 rows | write | 29.4 ms | 30.3 ms | +3% |
   | 8192 rows | write + read | 58.2 ms | 58.0 ms | no change |
   
   On disk, the 128-row input goes from 8,192 IPC batches (88.5 MB) to 128 (72.0 MB, **−19%**). With 8192-row input, both versions write the same files.
   
   <details>
   <summary><code>benchmarks/benches/passthrough_shuffle.rs</code></summary>
   
   ```rust
   //! Criterion benchmarks for the passthrough shuffle writer.
   //!
   //! The passthrough writer (`ShuffleWriterExec`) writes each child partition
   //! to its own Arrow IPC file. These benchmarks feed it the same rows either
   //! as many small batches (as produced after a selective filter) or as
   //! `batch_size` batches, and measure the write alone and the write followed
   //! by reading every file back, as a reducer would.
   
   use std::fs::File;
   use std::io::BufReader;
   use std::sync::Arc;
   
   use ballista_core::execution_plans::ShuffleWriterExec;
   use criterion::{Criterion, criterion_group, criterion_main};
   use datafusion::arrow::array::{Array, Float64Array, Int64Array, StringArray};
   use datafusion::arrow::datatypes::{DataType, Field, Schema, SchemaRef};
   use datafusion::arrow::ipc::reader::StreamReader;
   use datafusion::arrow::record_batch::RecordBatch;
   use datafusion::datasource::memory::MemorySourceConfig;
   use datafusion::datasource::source::DataSourceExec;
   use datafusion::physical_plan::ExecutionPlan;
   use datafusion::prelude::SessionContext;
   use futures::TryStreamExt;
   use rand::RngExt;
   use tempfile::TempDir;
   use tokio::runtime::Runtime;
   
   const NUM_PARTITIONS: usize = 8;
   const ROWS_PER_PARTITION: usize = 131_072;
   
   fn build_schema() -> SchemaRef {
       Arc::new(Schema::new(vec![
           Field::new("i0", DataType::Int64, false),
           Field::new("i1", DataType::Int64, false),
           Field::new("i2", DataType::Int64, false),
           Field::new("i3", DataType::Int64, false),
           Field::new("f0", DataType::Float64, false),
           Field::new("f1", DataType::Float64, false),
           Field::new("s0", DataType::Utf8, false),
           Field::new("s1", DataType::Utf8, false),
       ]))
   }
   
   fn build_batch(schema: &SchemaRef, rows: usize) -> RecordBatch {
       let mut rng = rand::rng();
       let mut columns: Vec<Arc<dyn Array>> = Vec::with_capacity(8);
       for _ in 0..4 {
           let vals: Vec<i64> = (0..rows).map(|_| rng.random()).collect();
           columns.push(Arc::new(Int64Array::from(vals)));
       }
       for _ in 0..2 {
           let vals: Vec<f64> = (0..rows).map(|_| rng.random()).collect();
           columns.push(Arc::new(Float64Array::from(vals)));
       }
       for _ in 0..2 {
           let vals: Vec<String> = (0..rows)
               .map(|_| format!("value-{}", rng.random_range(0..1_000_000)))
               .collect();
           columns.push(Arc::new(StringArray::from(vals)));
       }
       RecordBatch::try_new(schema.clone(), columns).unwrap()
   }
   
   /// `NUM_PARTITIONS` partitions of the same `ROWS_PER_PARTITION` distinct rows,
   /// delivered as slices of `rows_per_batch` rows, so every input size writes
   /// identical data.
   fn create_input(rows: &RecordBatch, rows_per_batch: usize) -> Arc<dyn ExecutionPlan> {
       let schema = rows.schema();
       let partition: Vec<RecordBatch> = (0..ROWS_PER_PARTITION)
           .step_by(rows_per_batch)
           .map(|offset| rows.slice(offset, rows_per_batch))
           .collect();
       let partitions = vec![partition; NUM_PARTITIONS];
       let memory_source =
           Arc::new(MemorySourceConfig::try_new(&partitions, schema, None).unwrap());
       Arc::new(DataSourceExec::new(memory_source))
   }
   
   /// Runs the writer over every output partition and returns the file paths.
   fn run_write(rt: &Runtime, input: Arc<dyn ExecutionPlan>, work_dir: &str) -> Vec<String> {
       let writer = Arc::new(
           ShuffleWriterExec::try_new("bench_job".into(), 1, input, work_dir.to_string())
               .unwrap(),
       );
       let task_ctx = SessionContext::new().task_ctx();
       rt.block_on(async {
           let streams = (0..NUM_PARTITIONS)
               .map(|p| writer.execute(p, task_ctx.clone()).unwrap())
               .map(|s| s.try_collect::<Vec<_>>());
           let summaries = futures::future::try_join_all(streams).await.unwrap();
           summaries
               .iter()
               .flatten()
               .flat_map(|b| {
                   let paths = b.column(1).as_any().downcast_ref::<StringArray>().unwrap();
                   paths
                       .iter()
                       .map(|p| p.unwrap().to_string())
                       .collect::<Vec<_>>()
               })
               .collect()
       })
   }
   
   /// Reads every file back and returns `(rows, ipc_batches)`.
   fn read_back(paths: &[String]) -> (usize, usize) {
       let mut rows = 0;
       let mut batches = 0;
       for path in paths {
           let file = BufReader::new(File::open(path).unwrap());
           for batch in StreamReader::try_new(file, None).unwrap() {
               rows += batch.unwrap().num_rows();
               batches += 1;
           }
       }
       (rows, batches)
   }
   
   fn bench_passthrough_shuffle(c: &mut Criterion) {
       let rt = Runtime::new().unwrap();
       let work_dir = TempDir::new().unwrap();
       let dir = work_dir.path().to_str().unwrap();
       let rows = build_batch(&build_schema(), ROWS_PER_PARTITION);
   
       for (name, rows_per_batch) in [("128_row_batches", 128), ("8192_row_batches", 8192)] {
           let input = create_input(&rows, rows_per_batch);
   
           let paths = run_write(&rt, input.clone(), dir);
           let bytes: u64 = paths
               .iter()
               .map(|p| std::fs::metadata(p).unwrap().len())
               .sum();
           let (rows, batches) = read_back(&paths);
           println!(
               "{name}: {rows} rows, {batches} IPC batches, {bytes} bytes on disk across {} files",
               paths.len()
           );
   
           let mut group = c.benchmark_group("passthrough_shuffle");
           group.sample_size(20);
           group.bench_function(format!("write/{name}"), |b| {
               b.iter(|| run_write(&rt, input.clone(), dir));
           });
           group.bench_function(format!("write_and_read/{name}"), |b| {
               b.iter(|| read_back(&run_write(&rt, input.clone(), dir)));
           });
           group.finish();
       }
   }
   
   criterion_group!(benches, bench_passthrough_shuffle);
   criterion_main!(benches);
   ```
   
   Register it in `benchmarks/Cargo.toml` with `[[bench]] harness = false, name = "passthrough_shuffle"`, then run `TMPDIR=/dev/shm cargo bench -p ballista-benchmarks --bench passthrough_shuffle`.
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf(shuffle-writer): coalesce small batches before writing passthrough shuffle files [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #2518:
URL: https://github.com/apache/datafusion-ballista/pull/2518#issuecomment-5941481072

   > > Requesting changes for two memory-scalability issues in the new coalescing path. The functional behavior and focused tests look good, but aggregate buffering is not bounded for high output-partition counts, and one input sequence can exceed the stated per-partition bound.
   > 
   > Sorry for the trouble, I'll slow down and re-review. 👀
   
   We went through similar experiments in Comet. We got better performance with coalescers per output partition but it didn't scale with high partition counts.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org