You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Rich-T-kid (via GitHub)" <gi...@apache.org> on 2026/09/09 18:25:33 UTC
[PR] Rich t kid/introduce repartion exec benchmarks [datafusion]
Rich-T-kid opened a new pull request, #25128:
URL: https://github.com/apache/datafusion/pull/25128
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #25127.
## Rationale for this change
RepartitionExec is one of the most frequently executed operators in DataFusion, every parallel query goes through it, but there were no dedicated benchmarks for it.
also see #25127
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
Please explain the problem you are trying to solve in terms of the user-visible
behavior, rather than the implementation.
For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the
implementation. "COUNT(DISTINCT) returns wrong results when the column contains
nulls" is the user-visible problem.
-->
## What changes are included in this PR?
Adds `datafusion/physical-plan/benches/repartition.rs` with two categories of benchmarks:
BatchPartitioner microbenchmarks (no async, pure CPU routing cost):
- `hash_partitioner/partition_count` — sweeps power-of-two and non-power-of-two counts (8, 10, 16, 32, 64, 100, 128), exercising both branches of StrengthReducedU64 (bitmask vs. reciprocal multiply)
- `hash_partitioner/key_type` — compares Int32, Int64, and Utf8View hashing cost at 32 partitions
- `hash_partitioner/key_count` — 1, 2, and 3 composite key columns at 32 partitions, showing per-key overhead
- `round_robin_partitioner/partition_count` — trivial routing baseline for isolating channel overhead from hash cost
End-to-end RepartitionExec benchmarks (full async path, 5M rows):
- `repartition_exec/hash_1_to_n` — 1 input → N outputs (4, 8, 16, 32) with hash partitioning
- `repartition_exec/round_robin_1_to_n` — same topology with round-robin, for direct comparison against hash
- `repartition_exec/hash_n_to_n` — N inputs → N outputs (4, 8, 16), the typical concurrent-producer scenario
- `repartition_exec/hash_m_to_n` — asymmetric topologies: consolidation (16→4, 16→8, 32→8) and fan-out (4→16, 8→16, 8→32)
<!--
There is no need to duplicate the description in the issue here, but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## What is the testing strategy for this PR?
n/a
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
Briefly describe how this PR is tested, and point to the specific tests you added. For example: 'This new feature is covered by the `sqllogictest` cases added in `foo.slt`'.
If this PR does not add tests, explain why. For example, if the change is already covered by existing tests, please mention it.
You should also check the `codecov` bot reply on this PR to confirm the changed code is exercised.
-->
## Are there any user-facing changes?
no
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] benchmarks: introduce RepartitionExec benchmarks [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid closed pull request #25128: benchmarks: introduce RepartitionExec benchmarks
URL: https://github.com/apache/datafusion/pull/25128
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] benchmarks: introduce RepartitionExec benchmarks [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #25128:
URL: https://github.com/apache/datafusion/pull/25128#discussion_r3974354591
##########
datafusion/physical-plan/benches/repartition.rs:
##########
@@ -0,0 +1,540 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint::black_box;
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int32Array, Int64Array, RecordBatch, StringViewArray};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use criterion::{
+ BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main,
+};
+use datafusion_execution::TaskContext;
+use datafusion_physical_expr::PhysicalExpr;
+use datafusion_physical_expr::expressions::col;
+use datafusion_physical_plan::metrics::Time;
+use datafusion_physical_plan::repartition::{BatchPartitioner, RepartitionExec};
+use datafusion_physical_plan::test::TestMemoryExec;
+use datafusion_physical_plan::{Partitioning, collect};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+use tokio::runtime::Runtime;
+
+const BATCH_SIZE: usize = 8192;
+const SEED: u64 = 42;
+
+// Intentionally mixes powers-of-two (8, 16, 32, 64, 128) and non-powers-of-two
+// (10, 100) so both StrengthReducedU64 branches (bit-mask and reciprocal) are
+// exercised in every benchmark run.
+const PARTITION_COUNTS: &[usize] = &[8, 10, 16, 32, 64, 100, 128];
+
+fn make_i64_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_i32_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i32> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i32))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int32Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_utf8view_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<String> = (0..num_rows)
+ .map(|_| format!("key_{:08}", rng.random_range(0..1_000_000usize)))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(StringViewArray::from_iter_values(
+ keys.iter().map(String::as_str),
+ )) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_multi_key_batch(
+ schema: &SchemaRef,
+ num_keys: usize,
+ num_rows: usize,
+) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let mut columns: Vec<ArrayRef> = (0..num_keys)
+ .map(|_| {
+ let v: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ Arc::new(Int64Array::from(v)) as ArrayRef
+ })
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ columns.push(Arc::new(Int64Array::from(vals)) as ArrayRef);
+ RecordBatch::try_new(Arc::clone(schema), columns).unwrap()
+}
+
+/// Build `num_partitions` input partition slots, each holding `rows_per_partition / batch_size`
+/// batches of random Int64 data.
+fn make_partitioned_input(
+ num_partitions: usize,
+ rows_per_partition: usize,
+) -> (Vec<Vec<RecordBatch>>, SchemaRef) {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let num_batches = (rows_per_partition + BATCH_SIZE - 1) / BATCH_SIZE;
+ let partitions = (0..num_partitions)
+ .map(|_| {
+ (0..num_batches)
+ .map(|_| {
+ let keys: Vec<i64> = (0..BATCH_SIZE)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..BATCH_SIZE as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+ })
+ .collect()
+ })
+ .collect();
+ (partitions, schema)
+}
+
+/// Hash routing at varying partition counts.
+///
+/// Power-of-two counts (8, 16, 32 …) use a bitmask inside `StrengthReducedU64`;
+/// non-powers (10, 100 …) use a reciprocal multiply. Both paths are included so
+/// the difference in routing cost is visible.
Review Comment:
can remove this. `PARTITION_COUNTS` already mentions this
##########
datafusion/physical-plan/benches/repartition.rs:
##########
@@ -0,0 +1,540 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint::black_box;
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int32Array, Int64Array, RecordBatch, StringViewArray};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use criterion::{
+ BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main,
+};
+use datafusion_execution::TaskContext;
+use datafusion_physical_expr::PhysicalExpr;
+use datafusion_physical_expr::expressions::col;
+use datafusion_physical_plan::metrics::Time;
+use datafusion_physical_plan::repartition::{BatchPartitioner, RepartitionExec};
+use datafusion_physical_plan::test::TestMemoryExec;
+use datafusion_physical_plan::{Partitioning, collect};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+use tokio::runtime::Runtime;
+
+const BATCH_SIZE: usize = 8192;
+const SEED: u64 = 42;
+
+// Intentionally mixes powers-of-two (8, 16, 32, 64, 128) and non-powers-of-two
+// (10, 100) so both StrengthReducedU64 branches (bit-mask and reciprocal) are
+// exercised in every benchmark run.
+const PARTITION_COUNTS: &[usize] = &[8, 10, 16, 32, 64, 100, 128];
+
+fn make_i64_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_i32_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i32> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i32))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int32Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_utf8view_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<String> = (0..num_rows)
+ .map(|_| format!("key_{:08}", rng.random_range(0..1_000_000usize)))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(StringViewArray::from_iter_values(
+ keys.iter().map(String::as_str),
+ )) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_multi_key_batch(
+ schema: &SchemaRef,
+ num_keys: usize,
+ num_rows: usize,
+) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let mut columns: Vec<ArrayRef> = (0..num_keys)
+ .map(|_| {
+ let v: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ Arc::new(Int64Array::from(v)) as ArrayRef
+ })
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ columns.push(Arc::new(Int64Array::from(vals)) as ArrayRef);
+ RecordBatch::try_new(Arc::clone(schema), columns).unwrap()
+}
+
+/// Build `num_partitions` input partition slots, each holding `rows_per_partition / batch_size`
+/// batches of random Int64 data.
+fn make_partitioned_input(
+ num_partitions: usize,
+ rows_per_partition: usize,
+) -> (Vec<Vec<RecordBatch>>, SchemaRef) {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let num_batches = (rows_per_partition + BATCH_SIZE - 1) / BATCH_SIZE;
+ let partitions = (0..num_partitions)
+ .map(|_| {
+ (0..num_batches)
+ .map(|_| {
+ let keys: Vec<i64> = (0..BATCH_SIZE)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..BATCH_SIZE as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+ })
+ .collect()
+ })
+ .collect();
+ (partitions, schema)
+}
+
+/// Hash routing at varying partition counts.
+///
+/// Power-of-two counts (8, 16, 32 …) use a bitmask inside `StrengthReducedU64`;
+/// non-powers (10, 100 …) use a reciprocal multiply. Both paths are included so
+/// the difference in routing cost is visible.
+fn bench_hash_partitioner_partition_count(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/partition_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+
+ for &n in PARTITION_COUNTS {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ let mut partitioner = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ n,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ partitioner
+ .partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+ group.finish();
+}
+
+/// Hash routing cost per key column type at a fixed 32 output partitions.
+fn bench_hash_partitioner_key_types(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/key_type");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+ const N: usize = 32;
+
+ // Int32
Review Comment:
we dont need these comments
##########
datafusion/physical-plan/benches/repartition.rs:
##########
@@ -0,0 +1,540 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint::black_box;
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int32Array, Int64Array, RecordBatch, StringViewArray};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use criterion::{
+ BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main,
+};
+use datafusion_execution::TaskContext;
+use datafusion_physical_expr::PhysicalExpr;
+use datafusion_physical_expr::expressions::col;
+use datafusion_physical_plan::metrics::Time;
+use datafusion_physical_plan::repartition::{BatchPartitioner, RepartitionExec};
+use datafusion_physical_plan::test::TestMemoryExec;
+use datafusion_physical_plan::{Partitioning, collect};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+use tokio::runtime::Runtime;
+
+const BATCH_SIZE: usize = 8192;
+const SEED: u64 = 42;
+
+// Intentionally mixes powers-of-two (8, 16, 32, 64, 128) and non-powers-of-two
+// (10, 100) so both StrengthReducedU64 branches (bit-mask and reciprocal) are
+// exercised in every benchmark run.
+const PARTITION_COUNTS: &[usize] = &[8, 10, 16, 32, 64, 100, 128];
+
+fn make_i64_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_i32_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i32> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i32))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int32Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_utf8view_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<String> = (0..num_rows)
+ .map(|_| format!("key_{:08}", rng.random_range(0..1_000_000usize)))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(StringViewArray::from_iter_values(
+ keys.iter().map(String::as_str),
+ )) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_multi_key_batch(
+ schema: &SchemaRef,
+ num_keys: usize,
+ num_rows: usize,
+) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let mut columns: Vec<ArrayRef> = (0..num_keys)
+ .map(|_| {
+ let v: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ Arc::new(Int64Array::from(v)) as ArrayRef
+ })
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ columns.push(Arc::new(Int64Array::from(vals)) as ArrayRef);
+ RecordBatch::try_new(Arc::clone(schema), columns).unwrap()
+}
+
+/// Build `num_partitions` input partition slots, each holding `rows_per_partition / batch_size`
+/// batches of random Int64 data.
+fn make_partitioned_input(
+ num_partitions: usize,
+ rows_per_partition: usize,
+) -> (Vec<Vec<RecordBatch>>, SchemaRef) {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let num_batches = (rows_per_partition + BATCH_SIZE - 1) / BATCH_SIZE;
+ let partitions = (0..num_partitions)
+ .map(|_| {
+ (0..num_batches)
+ .map(|_| {
+ let keys: Vec<i64> = (0..BATCH_SIZE)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..BATCH_SIZE as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
Review Comment:
should add non primitive types for columns here
##########
datafusion/physical-plan/benches/repartition.rs:
##########
@@ -0,0 +1,540 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint::black_box;
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int32Array, Int64Array, RecordBatch, StringViewArray};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use criterion::{
+ BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main,
+};
+use datafusion_execution::TaskContext;
+use datafusion_physical_expr::PhysicalExpr;
+use datafusion_physical_expr::expressions::col;
+use datafusion_physical_plan::metrics::Time;
+use datafusion_physical_plan::repartition::{BatchPartitioner, RepartitionExec};
+use datafusion_physical_plan::test::TestMemoryExec;
+use datafusion_physical_plan::{Partitioning, collect};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+use tokio::runtime::Runtime;
+
+const BATCH_SIZE: usize = 8192;
+const SEED: u64 = 42;
+
+// Intentionally mixes powers-of-two (8, 16, 32, 64, 128) and non-powers-of-two
+// (10, 100) so both StrengthReducedU64 branches (bit-mask and reciprocal) are
+// exercised in every benchmark run.
+const PARTITION_COUNTS: &[usize] = &[8, 10, 16, 32, 64, 100, 128];
+
+fn make_i64_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_i32_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i32> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i32))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int32Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_utf8view_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<String> = (0..num_rows)
+ .map(|_| format!("key_{:08}", rng.random_range(0..1_000_000usize)))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(StringViewArray::from_iter_values(
+ keys.iter().map(String::as_str),
+ )) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_multi_key_batch(
+ schema: &SchemaRef,
+ num_keys: usize,
+ num_rows: usize,
+) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let mut columns: Vec<ArrayRef> = (0..num_keys)
+ .map(|_| {
+ let v: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ Arc::new(Int64Array::from(v)) as ArrayRef
+ })
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ columns.push(Arc::new(Int64Array::from(vals)) as ArrayRef);
+ RecordBatch::try_new(Arc::clone(schema), columns).unwrap()
+}
+
+/// Build `num_partitions` input partition slots, each holding `rows_per_partition / batch_size`
+/// batches of random Int64 data.
+fn make_partitioned_input(
+ num_partitions: usize,
+ rows_per_partition: usize,
+) -> (Vec<Vec<RecordBatch>>, SchemaRef) {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let num_batches = (rows_per_partition + BATCH_SIZE - 1) / BATCH_SIZE;
+ let partitions = (0..num_partitions)
+ .map(|_| {
+ (0..num_batches)
+ .map(|_| {
+ let keys: Vec<i64> = (0..BATCH_SIZE)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..BATCH_SIZE as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+ })
+ .collect()
+ })
+ .collect();
+ (partitions, schema)
+}
+
+/// Hash routing at varying partition counts.
+///
+/// Power-of-two counts (8, 16, 32 …) use a bitmask inside `StrengthReducedU64`;
+/// non-powers (10, 100 …) use a reciprocal multiply. Both paths are included so
+/// the difference in routing cost is visible.
+fn bench_hash_partitioner_partition_count(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/partition_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+
+ for &n in PARTITION_COUNTS {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ let mut partitioner = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ n,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ partitioner
+ .partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+ group.finish();
+}
+
+/// Hash routing cost per key column type at a fixed 32 output partitions.
+fn bench_hash_partitioner_key_types(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/key_type");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+ const N: usize = 32;
+
+ // Int32
+ {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int32, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i32_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+ group.bench_function("int32", |b| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+
+ // Int64
+ {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+ group.bench_function("int64", |b| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+
+ // Utf8View
+ {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Utf8View, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_utf8view_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+ group.bench_function("utf8", |b| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+
+ group.finish();
+}
+
+/// Composite hash cost for 1, 2, and 3 key columns.
+///
+/// Each extra key column adds an independent hash pass over all rows, so this
+/// shows how multi-column GROUP BY / JOIN keys scale the routing cost.
+fn bench_hash_partitioner_key_count(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/key_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+ const N: usize = 32;
+
+ for num_keys in [1usize, 2, 3] {
+ let fields: Vec<Field> = (0..num_keys)
+ .map(|i| Field::new(format!("k{i}"), DataType::Int64, false))
+ .chain(std::iter::once(Field::new("val", DataType::Int64, false)))
+ .collect();
+ let schema = Arc::new(Schema::new(fields));
+ let batch = make_multi_key_batch(&schema, num_keys, BATCH_SIZE);
+ let key_exprs: Vec<Arc<dyn PhysicalExpr>> = (0..num_keys)
+ .map(|i| col(&format!("k{i}"), &schema).unwrap())
+ .collect();
+
+ group.bench_with_input(
+ BenchmarkId::from_parameter(num_keys),
+ &num_keys,
+ |b, _| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ key_exprs.clone(),
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ },
+ );
+ }
+ group.finish();
+}
+
+/// Round-robin routing at varying partition counts.
+///
+/// Round-robin does no hashing, it simply increments a counter. This is the
+/// baseline for the `partition()` interface overhead. Compare against
+/// `hash_partitioner/partition_count` to isolate pure hashing cost.
+fn bench_round_robin_partitioner(c: &mut Criterion) {
+ let mut group = c.benchmark_group("round_robin_partitioner/partition_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+
+ for &n in PARTITION_COUNTS {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ let mut partitioner =
+ BatchPartitioner::new_round_robin_partitioner(n, Time::default(), 0, 1);
+ b.iter(|| {
+ partitioner
+ .partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+ group.finish();
+}
+
+const E2E_ROWS: usize = 5_000_000;
+
+/// Hash repartition: 1 input partition → N output partitions.
+///
+/// Measures the full operator path: batch partitioning, channel sends,
+/// coalescing, memory reservation, and downstream collection.
Review Comment:
trim
##########
datafusion/physical-plan/benches/repartition.rs:
##########
@@ -0,0 +1,540 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint::black_box;
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int32Array, Int64Array, RecordBatch, StringViewArray};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use criterion::{
+ BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main,
+};
+use datafusion_execution::TaskContext;
+use datafusion_physical_expr::PhysicalExpr;
+use datafusion_physical_expr::expressions::col;
+use datafusion_physical_plan::metrics::Time;
+use datafusion_physical_plan::repartition::{BatchPartitioner, RepartitionExec};
+use datafusion_physical_plan::test::TestMemoryExec;
+use datafusion_physical_plan::{Partitioning, collect};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+use tokio::runtime::Runtime;
+
+const BATCH_SIZE: usize = 8192;
+const SEED: u64 = 42;
+
+// Intentionally mixes powers-of-two (8, 16, 32, 64, 128) and non-powers-of-two
+// (10, 100) so both StrengthReducedU64 branches (bit-mask and reciprocal) are
+// exercised in every benchmark run.
+const PARTITION_COUNTS: &[usize] = &[8, 10, 16, 32, 64, 100, 128];
+
+fn make_i64_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_i32_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i32> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i32))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int32Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_utf8view_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<String> = (0..num_rows)
+ .map(|_| format!("key_{:08}", rng.random_range(0..1_000_000usize)))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(StringViewArray::from_iter_values(
+ keys.iter().map(String::as_str),
+ )) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_multi_key_batch(
+ schema: &SchemaRef,
+ num_keys: usize,
+ num_rows: usize,
+) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let mut columns: Vec<ArrayRef> = (0..num_keys)
+ .map(|_| {
+ let v: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ Arc::new(Int64Array::from(v)) as ArrayRef
+ })
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ columns.push(Arc::new(Int64Array::from(vals)) as ArrayRef);
+ RecordBatch::try_new(Arc::clone(schema), columns).unwrap()
+}
+
+/// Build `num_partitions` input partition slots, each holding `rows_per_partition / batch_size`
+/// batches of random Int64 data.
+fn make_partitioned_input(
+ num_partitions: usize,
+ rows_per_partition: usize,
+) -> (Vec<Vec<RecordBatch>>, SchemaRef) {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let num_batches = (rows_per_partition + BATCH_SIZE - 1) / BATCH_SIZE;
+ let partitions = (0..num_partitions)
+ .map(|_| {
+ (0..num_batches)
+ .map(|_| {
+ let keys: Vec<i64> = (0..BATCH_SIZE)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..BATCH_SIZE as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+ })
+ .collect()
+ })
+ .collect();
+ (partitions, schema)
+}
+
+/// Hash routing at varying partition counts.
+///
+/// Power-of-two counts (8, 16, 32 …) use a bitmask inside `StrengthReducedU64`;
+/// non-powers (10, 100 …) use a reciprocal multiply. Both paths are included so
+/// the difference in routing cost is visible.
+fn bench_hash_partitioner_partition_count(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/partition_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+
+ for &n in PARTITION_COUNTS {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ let mut partitioner = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ n,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ partitioner
+ .partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+ group.finish();
+}
+
+/// Hash routing cost per key column type at a fixed 32 output partitions.
+fn bench_hash_partitioner_key_types(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/key_type");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+ const N: usize = 32;
+
+ // Int32
+ {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int32, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i32_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+ group.bench_function("int32", |b| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+
+ // Int64
+ {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+ group.bench_function("int64", |b| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+
+ // Utf8View
+ {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Utf8View, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_utf8view_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+ group.bench_function("utf8", |b| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+
+ group.finish();
+}
+
+/// Composite hash cost for 1, 2, and 3 key columns.
+///
+/// Each extra key column adds an independent hash pass over all rows, so this
+/// shows how multi-column GROUP BY / JOIN keys scale the routing cost.
+fn bench_hash_partitioner_key_count(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/key_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+ const N: usize = 32;
+
+ for num_keys in [1usize, 2, 3] {
+ let fields: Vec<Field> = (0..num_keys)
+ .map(|i| Field::new(format!("k{i}"), DataType::Int64, false))
+ .chain(std::iter::once(Field::new("val", DataType::Int64, false)))
+ .collect();
+ let schema = Arc::new(Schema::new(fields));
+ let batch = make_multi_key_batch(&schema, num_keys, BATCH_SIZE);
+ let key_exprs: Vec<Arc<dyn PhysicalExpr>> = (0..num_keys)
+ .map(|i| col(&format!("k{i}"), &schema).unwrap())
+ .collect();
+
+ group.bench_with_input(
+ BenchmarkId::from_parameter(num_keys),
+ &num_keys,
+ |b, _| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ key_exprs.clone(),
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ },
+ );
+ }
+ group.finish();
+}
+
+/// Round-robin routing at varying partition counts.
+///
+/// Round-robin does no hashing, it simply increments a counter. This is the
+/// baseline for the `partition()` interface overhead. Compare against
+/// `hash_partitioner/partition_count` to isolate pure hashing cost.
+fn bench_round_robin_partitioner(c: &mut Criterion) {
+ let mut group = c.benchmark_group("round_robin_partitioner/partition_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+
+ for &n in PARTITION_COUNTS {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ let mut partitioner =
+ BatchPartitioner::new_round_robin_partitioner(n, Time::default(), 0, 1);
+ b.iter(|| {
+ partitioner
+ .partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+ group.finish();
+}
+
+const E2E_ROWS: usize = 5_000_000;
+
+/// Hash repartition: 1 input partition → N output partitions.
+///
+/// Measures the full operator path: batch partitioning, channel sends,
+/// coalescing, memory reservation, and downstream collection.
+fn bench_repartition_exec_hash_1_to_n(c: &mut Criterion) {
+ let rt = Runtime::new().unwrap();
+ let task_ctx = Arc::new(TaskContext::default());
+ let mut group = c.benchmark_group("repartition_exec/hash_1_to_n");
+ group.throughput(Throughput::Elements(E2E_ROWS as u64));
+
+ let (partitions, schema) = make_partitioned_input(1, E2E_ROWS);
+ let key_expr = col("key", &schema).unwrap();
+
+ for &n in &[4usize, 8, 16, 32] {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ b.iter_batched(
+ || {
+ let input =
+ TestMemoryExec::try_new_exec(&partitions, schema.clone(), None)
+ .unwrap();
+ Arc::new(
+ RepartitionExec::try_new(
+ input,
+ Partitioning::Hash(vec![Arc::clone(&key_expr)], n),
+ )
+ .unwrap(),
+ )
+ },
+ |plan| {
+ rt.block_on(async {
+ collect(plan, task_ctx.clone()).await.unwrap();
+ });
+ },
+ BatchSize::LargeInput,
+ );
+ });
+ }
+ group.finish();
+}
+
+/// Round-robin repartition: 1 input partition → N output partitions.
+///
+/// Compare against `hash_1_to_n` to isolate the cost of hash computation
+/// versus channel/coalescing overhead that both modes share.
+fn bench_repartition_exec_round_robin_1_to_n(c: &mut Criterion) {
+ let rt = Runtime::new().unwrap();
+ let task_ctx = Arc::new(TaskContext::default());
+ let mut group = c.benchmark_group("repartition_exec/round_robin_1_to_n");
+ group.throughput(Throughput::Elements(E2E_ROWS as u64));
+
+ let (partitions, schema) = make_partitioned_input(1, E2E_ROWS);
+
+ for &n in &[4usize, 8, 16, 32] {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ b.iter_batched(
+ || {
+ let input =
+ TestMemoryExec::try_new_exec(&partitions, schema.clone(), None)
+ .unwrap();
+ Arc::new(
+ RepartitionExec::try_new(input, Partitioning::RoundRobinBatch(n))
+ .unwrap(),
+ )
+ },
+ |plan| {
+ rt.block_on(async {
+ collect(plan, task_ctx.clone()).await.unwrap();
+ });
+ },
+ BatchSize::LargeInput,
+ );
+ });
+ }
+ group.finish();
+}
Review Comment:
might be better to combine this with the hash partion function above & loop through the two different partition approaches. less code bloat
##########
datafusion/physical-plan/benches/repartition.rs:
##########
@@ -0,0 +1,540 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::hint::black_box;
+use std::sync::Arc;
+
+use arrow::array::{ArrayRef, Int32Array, Int64Array, RecordBatch, StringViewArray};
+use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
+use criterion::{
+ BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main,
+};
+use datafusion_execution::TaskContext;
+use datafusion_physical_expr::PhysicalExpr;
+use datafusion_physical_expr::expressions::col;
+use datafusion_physical_plan::metrics::Time;
+use datafusion_physical_plan::repartition::{BatchPartitioner, RepartitionExec};
+use datafusion_physical_plan::test::TestMemoryExec;
+use datafusion_physical_plan::{Partitioning, collect};
+use rand::rngs::StdRng;
+use rand::{Rng, SeedableRng};
+use tokio::runtime::Runtime;
+
+const BATCH_SIZE: usize = 8192;
+const SEED: u64 = 42;
+
+// Intentionally mixes powers-of-two (8, 16, 32, 64, 128) and non-powers-of-two
+// (10, 100) so both StrengthReducedU64 branches (bit-mask and reciprocal) are
+// exercised in every benchmark run.
+const PARTITION_COUNTS: &[usize] = &[8, 10, 16, 32, 64, 100, 128];
+
+fn make_i64_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_i32_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<i32> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i32))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(Int32Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_utf8view_batch(schema: &SchemaRef, num_rows: usize) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let keys: Vec<String> = (0..num_rows)
+ .map(|_| format!("key_{:08}", rng.random_range(0..1_000_000usize)))
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(schema),
+ vec![
+ Arc::new(StringViewArray::from_iter_values(
+ keys.iter().map(String::as_str),
+ )) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+}
+
+fn make_multi_key_batch(
+ schema: &SchemaRef,
+ num_keys: usize,
+ num_rows: usize,
+) -> RecordBatch {
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let mut columns: Vec<ArrayRef> = (0..num_keys)
+ .map(|_| {
+ let v: Vec<i64> = (0..num_rows)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ Arc::new(Int64Array::from(v)) as ArrayRef
+ })
+ .collect();
+ let vals: Vec<i64> = (0..num_rows as i64).collect();
+ columns.push(Arc::new(Int64Array::from(vals)) as ArrayRef);
+ RecordBatch::try_new(Arc::clone(schema), columns).unwrap()
+}
+
+/// Build `num_partitions` input partition slots, each holding `rows_per_partition / batch_size`
+/// batches of random Int64 data.
+fn make_partitioned_input(
+ num_partitions: usize,
+ rows_per_partition: usize,
+) -> (Vec<Vec<RecordBatch>>, SchemaRef) {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let mut rng = StdRng::seed_from_u64(SEED);
+ let num_batches = (rows_per_partition + BATCH_SIZE - 1) / BATCH_SIZE;
+ let partitions = (0..num_partitions)
+ .map(|_| {
+ (0..num_batches)
+ .map(|_| {
+ let keys: Vec<i64> = (0..BATCH_SIZE)
+ .map(|_| rng.random_range(0..1_000_000i64))
+ .collect();
+ let vals: Vec<i64> = (0..BATCH_SIZE as i64).collect();
+ RecordBatch::try_new(
+ Arc::clone(&schema),
+ vec![
+ Arc::new(Int64Array::from(keys)) as ArrayRef,
+ Arc::new(Int64Array::from(vals)),
+ ],
+ )
+ .unwrap()
+ })
+ .collect()
+ })
+ .collect();
+ (partitions, schema)
+}
+
+/// Hash routing at varying partition counts.
+///
+/// Power-of-two counts (8, 16, 32 …) use a bitmask inside `StrengthReducedU64`;
+/// non-powers (10, 100 …) use a reciprocal multiply. Both paths are included so
+/// the difference in routing cost is visible.
+fn bench_hash_partitioner_partition_count(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/partition_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+
+ for &n in PARTITION_COUNTS {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ let mut partitioner = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ n,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ partitioner
+ .partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+ group.finish();
+}
+
+/// Hash routing cost per key column type at a fixed 32 output partitions.
+fn bench_hash_partitioner_key_types(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/key_type");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+ const N: usize = 32;
+
+ // Int32
+ {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int32, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i32_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+ group.bench_function("int32", |b| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+
+ // Int64
+ {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+ group.bench_function("int64", |b| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+
+ // Utf8View
+ {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Utf8View, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_utf8view_batch(&schema, BATCH_SIZE);
+ let key_expr = col("key", &schema).unwrap();
+ group.bench_function("utf8", |b| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ vec![Arc::clone(&key_expr)],
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+
+ group.finish();
+}
+
+/// Composite hash cost for 1, 2, and 3 key columns.
+///
+/// Each extra key column adds an independent hash pass over all rows, so this
+/// shows how multi-column GROUP BY / JOIN keys scale the routing cost.
+fn bench_hash_partitioner_key_count(c: &mut Criterion) {
+ let mut group = c.benchmark_group("hash_partitioner/key_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+ const N: usize = 32;
+
+ for num_keys in [1usize, 2, 3] {
+ let fields: Vec<Field> = (0..num_keys)
+ .map(|i| Field::new(format!("k{i}"), DataType::Int64, false))
+ .chain(std::iter::once(Field::new("val", DataType::Int64, false)))
+ .collect();
+ let schema = Arc::new(Schema::new(fields));
+ let batch = make_multi_key_batch(&schema, num_keys, BATCH_SIZE);
+ let key_exprs: Vec<Arc<dyn PhysicalExpr>> = (0..num_keys)
+ .map(|i| col(&format!("k{i}"), &schema).unwrap())
+ .collect();
+
+ group.bench_with_input(
+ BenchmarkId::from_parameter(num_keys),
+ &num_keys,
+ |b, _| {
+ let mut p = BatchPartitioner::new_hash_partitioner(
+ key_exprs.clone(),
+ N,
+ Time::default(),
+ )
+ .unwrap();
+ b.iter(|| {
+ p.partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ },
+ );
+ }
+ group.finish();
+}
+
+/// Round-robin routing at varying partition counts.
+///
+/// Round-robin does no hashing, it simply increments a counter. This is the
+/// baseline for the `partition()` interface overhead. Compare against
+/// `hash_partitioner/partition_count` to isolate pure hashing cost.
+fn bench_round_robin_partitioner(c: &mut Criterion) {
+ let mut group = c.benchmark_group("round_robin_partitioner/partition_count");
+ group.throughput(Throughput::Elements(BATCH_SIZE as u64));
+
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("key", DataType::Int64, false),
+ Field::new("val", DataType::Int64, false),
+ ]));
+ let batch = make_i64_batch(&schema, BATCH_SIZE);
+
+ for &n in PARTITION_COUNTS {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ let mut partitioner =
+ BatchPartitioner::new_round_robin_partitioner(n, Time::default(), 0, 1);
+ b.iter(|| {
+ partitioner
+ .partition(batch.clone(), |p, b| {
+ black_box((p, b));
+ Ok(())
+ })
+ .unwrap();
+ });
+ });
+ }
+ group.finish();
+}
+
+const E2E_ROWS: usize = 5_000_000;
+
+/// Hash repartition: 1 input partition → N output partitions.
+///
+/// Measures the full operator path: batch partitioning, channel sends,
+/// coalescing, memory reservation, and downstream collection.
+fn bench_repartition_exec_hash_1_to_n(c: &mut Criterion) {
+ let rt = Runtime::new().unwrap();
+ let task_ctx = Arc::new(TaskContext::default());
+ let mut group = c.benchmark_group("repartition_exec/hash_1_to_n");
+ group.throughput(Throughput::Elements(E2E_ROWS as u64));
+
+ let (partitions, schema) = make_partitioned_input(1, E2E_ROWS);
+ let key_expr = col("key", &schema).unwrap();
+
+ for &n in &[4usize, 8, 16, 32] {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ b.iter_batched(
+ || {
+ let input =
+ TestMemoryExec::try_new_exec(&partitions, schema.clone(), None)
+ .unwrap();
+ Arc::new(
+ RepartitionExec::try_new(
+ input,
+ Partitioning::Hash(vec![Arc::clone(&key_expr)], n),
+ )
+ .unwrap(),
+ )
+ },
+ |plan| {
+ rt.block_on(async {
+ collect(plan, task_ctx.clone()).await.unwrap();
+ });
+ },
+ BatchSize::LargeInput,
+ );
+ });
+ }
+ group.finish();
+}
+
+/// Round-robin repartition: 1 input partition → N output partitions.
+///
+/// Compare against `hash_1_to_n` to isolate the cost of hash computation
+/// versus channel/coalescing overhead that both modes share.
+fn bench_repartition_exec_round_robin_1_to_n(c: &mut Criterion) {
+ let rt = Runtime::new().unwrap();
+ let task_ctx = Arc::new(TaskContext::default());
+ let mut group = c.benchmark_group("repartition_exec/round_robin_1_to_n");
+ group.throughput(Throughput::Elements(E2E_ROWS as u64));
+
+ let (partitions, schema) = make_partitioned_input(1, E2E_ROWS);
+
+ for &n in &[4usize, 8, 16, 32] {
+ group.bench_with_input(BenchmarkId::from_parameter(n), &n, |b, &n| {
+ b.iter_batched(
+ || {
+ let input =
+ TestMemoryExec::try_new_exec(&partitions, schema.clone(), None)
+ .unwrap();
+ Arc::new(
+ RepartitionExec::try_new(input, Partitioning::RoundRobinBatch(n))
+ .unwrap(),
+ )
+ },
+ |plan| {
+ rt.block_on(async {
+ collect(plan, task_ctx.clone()).await.unwrap();
+ });
+ },
+ BatchSize::LargeInput,
+ );
+ });
+ }
+ group.finish();
+}
+
+/// Hash repartition: N input partitions → N output partitions.
+///
+/// Models the typical distributed query scenario where N concurrent producer
+/// tasks each hash-route their share of rows to N output channels. The
+/// contention on shared channels and coalescer locks shows up here but not
+/// in the 1→N benchmarks.
+fn bench_repartition_exec_hash_n_to_n(c: &mut Criterion) {
Review Comment:
same idea here, mapping N -> N input:output would be nice to also get a feel for how round robin performs here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] benchmarks: introduce RepartitionExec benchmarks [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25128:
URL: https://github.com/apache/datafusion/pull/25128#issuecomment-5609919711
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25128?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 81.80%. Comparing base ([`4048898`](https://app.codecov.io/gh/apache/datafusion/commit/40488988ad596c9b093ad60e1453430d803ce33c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2381196`](https://app.codecov.io/gh/apache/datafusion/commit/238119616fc39b91927668f603f4ae1edd61ebc4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25128 +/- ##
==========================================
- Coverage 81.80% 81.80% -0.01%
==========================================
Files 1130 1130
Lines 417754 417754
Branches 417754 417754
==========================================
- Hits 341754 341750 -4
- Misses 55875 55877 +2
- Partials 20125 20127 +2
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25128?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] benchmarks: introduce RepartitionExec benchmarks [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #25128:
URL: https://github.com/apache/datafusion/pull/25128#issuecomment-5606851710
for the most part range partition wasn't included. could be a nice follow up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] benchmarks: introduce RepartitionExec benchmarks [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #25128:
URL: https://github.com/apache/datafusion/pull/25128#issuecomment-5606757653
@Dandandan @alamb could you take a look. Should be very useful for driving #7001 optimizations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org