You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/04/13 14:19:50 UTC
(datafusion) branch gh-readonly-queue/main/pr-21533-d3cedb282ae34dd72a6ffa9fa95b425e6761b1dd created (now 0143dfe169)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch gh-readonly-queue/main/pr-21533-d3cedb282ae34dd72a6ffa9fa95b425e6761b1dd
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at 0143dfe169 physical_optimizer: preserve_file_partitions when num file groups < target_partitions (#21533)
This branch includes the following new commits:
new 0143dfe169 physical_optimizer: preserve_file_partitions when num file groups < target_partitions (#21533)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: physical_optimizer: preserve_file_partitions when num file groups < target_partitions (#21533)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch gh-readonly-queue/main/pr-21533-d3cedb282ae34dd72a6ffa9fa95b425e6761b1dd
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit 0143dfe1692a7a56a72c736041d93797ef651642
Author: Jayant Shrivastava <ja...@datadoghq.com>
AuthorDate: Mon Apr 13 10:19:26 2026 -0400
physical_optimizer: preserve_file_partitions when num file groups < target_partitions (#21533)
## Rationale for this change
`datafusion.optimizer.preserve_file_partitions` would not actually
preserve the file partitions when the number of file groups is less than
the target_partitions. This is unexpected behavior. If a user wants to
preserve file partitions, it is because they want to avoid repartitions.
Before
```
ProjectionExec
AggregateExec: mode=FinalPartitioned, gby=[f_dkey, timestamp]
RepartitionExec: partitioning=Hash([f_dkey, timestamp], 4), input_partitions=3
AggregateExec: mode=Partial, gby=[f_dkey, timestamp]
DataSourceExec: file_groups=3, projection=[timestamp, value, f_dkey]
```
After
```
ProjectionExec
AggregateExec: mode=SinglePartitioned, gby=[f_dkey, timestamp]
DataSourceExec: file_groups=3, projection=[timestamp, value, f_dkey]
```
## What changes are included in this PR?
This change fixes that by updating 1 line in the `enforce_distribution`
optimizer rule.
## Are these changes tested?
Yes. In the `preserve_file_partitions` SLT.
## Are there any user-facing changes?
No.
---
.../physical-optimizer/src/enforce_distribution.rs | 8 +++-
.../test_files/preserve_file_partitioning.slt | 54 ++++++++++++++++++++++
2 files changed, 61 insertions(+), 1 deletion(-)
diff --git a/datafusion/physical-optimizer/src/enforce_distribution.rs b/datafusion/physical-optimizer/src/enforce_distribution.rs
index ea47c12030..87d0943610 100644
--- a/datafusion/physical-optimizer/src/enforce_distribution.rs
+++ b/datafusion/physical-optimizer/src/enforce_distribution.rs
@@ -1328,8 +1328,14 @@ pub fn ensure_distribution(
})
);
- let allow_subset_satisfy_partitioning = current_partitions
+ let allow_subset_satisfy_partitioning = (current_partitions
>= subset_satisfaction_threshold
+ // `preserve_file_partitions` exposes existing file-group
+ // partitioning to the optimizer. Respect it when the only
+ // reason to repartition would be to increase partition count
+ // beyond the preserved file-group count.
+ || (config.optimizer.preserve_file_partitions > 0
+ && current_partitions < target_partitions))
&& !is_partitioned_join
&& !requires_grouping_id;
diff --git a/datafusion/sqllogictest/test_files/preserve_file_partitioning.slt b/datafusion/sqllogictest/test_files/preserve_file_partitioning.slt
index 48d99625ea..175d7d90cd 100644
--- a/datafusion/sqllogictest/test_files/preserve_file_partitioning.slt
+++ b/datafusion/sqllogictest/test_files/preserve_file_partitioning.slt
@@ -698,6 +698,60 @@ A dev 772.4
B prod 614.4
C prod 2017.6
+##########
+# TEST 13: Partitioned Join where Number of File Groups is less than target_partitions
+# With preserve_file_partitions enabled, we should still avoid repartitioning
+##########
+
+statement ok
+set datafusion.execution.target_partitions = 4;
+
+statement ok
+set datafusion.optimizer.preserve_file_partitions = 1;
+
+query TT
+EXPLAIN SELECT f_dkey, timestamp,
+ COUNT(*), AVG(value)
+FROM fact_table
+GROUP BY f_dkey, timestamp;
+----
+logical_plan
+01)Projection: fact_table.f_dkey, fact_table.timestamp, count(Int64(1)) AS count(*), avg(fact_table.value)
+02)--Aggregate: groupBy=[[fact_table.f_dkey, fact_table.timestamp]], aggr=[[count(Int64(1)), avg(fact_table.value)]]
+03)----TableScan: fact_table projection=[timestamp, value, f_dkey]
+physical_plan
+01)ProjectionExec: expr=[f_dkey@0 as f_dkey, timestamp@1 as timestamp, count(Int64(1))@2 as count(*), avg(fact_table.value)@3 as avg(fact_table.value)]
+02)--AggregateExec: mode=SinglePartitioned, gby=[f_dkey@2 as f_dkey, timestamp@0 as timestamp], aggr=[count(Int64(1)), avg(fact_table.value)]
+03)----DataSourceExec: file_groups={3 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=A/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=B/data.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/preserve_file_partitioning/fact/f_dkey=C/data.parquet]]}, projection=[timestamp, value, f_dkey], file_type=parquet
+
+query TPIR rowsort
+SELECT f_dkey, timestamp,
+ COUNT(*), AVG(value)
+FROM fact_table
+GROUP BY f_dkey, timestamp;
+----
+A 2023-01-01T09:00:00 1 95.5
+A 2023-01-01T09:00:10 1 102.3
+A 2023-01-01T09:00:20 1 98.7
+A 2023-01-01T09:12:20 1 105.1
+A 2023-01-01T09:12:30 1 100
+A 2023-01-01T09:12:40 1 150
+A 2023-01-01T09:12:50 1 120.8
+B 2023-01-01T09:00:00 1 75.2
+B 2023-01-01T09:00:10 1 82.4
+B 2023-01-01T09:00:20 1 78.9
+B 2023-01-01T09:00:30 1 85.6
+B 2023-01-01T09:12:30 1 80
+B 2023-01-01T09:12:40 1 120
+B 2023-01-01T09:12:50 1 92.3
+C 2023-01-01T09:00:00 1 300.5
+C 2023-01-01T09:00:10 1 285.7
+C 2023-01-01T09:00:20 1 310.2
+C 2023-01-01T09:00:30 1 295.8
+C 2023-01-01T09:00:40 1 300
+C 2023-01-01T09:12:40 1 250
+C 2023-01-01T09:12:50 1 275.4
+
##########
# CLEANUP
##########
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org