You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "fvaleye (via GitHub)" <gi...@apache.org> on 2025/08/21 13:37:44 UTC

[PR] feat(datafusion): implement repartition node for DataFusion with [iceberg-rust]

fvaleye opened a new pull request, #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620

   ## Which issue does this PR close?
   - Closes #1543
   
   ## What changes are included in this PR?
   Implement a physical execution repartition node that determines the relevant DataFusion partitioning strategy based on the Iceberg table schema and metadata.
   - Implement hash partitioning for partitioned/bucketed tables
   - Use round-robin partitioning for unpartitioned tables
   - Support range distribution mode approximation via sort columns
   
   _Minor change: I created a new `schema_ref()` helper method._
   
   ## Are these changes tested?
   Yes, with unit tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2318443723


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();
+    let hash_exprs: Vec<Arc<dyn datafusion::physical_expr::PhysicalExpr>> = names_iter
+        .filter(|name| seen.insert(*name))
+        .map(|name| {
+            let idx = input_schema
+                .index_of(name)
+                .map_err(|e| datafusion::error::DataFusionError::Plan(e.to_string()))?;
+            Ok(Arc::new(Column::new(name, idx))

Review Comment:
   So we should resolve #1602 first?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2324722752


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   > DataFusion's Execution Partitioning: For the query optimizer to recognize that two data streams are partitioned compatibly (e.g., for a join), they must have the same canonical Partitioning scheme. Using an unstable order from a HashSet would break this, leading to inefficient plans.
   
   Thanks for the explaination, this use case makes sense to me. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2303784947


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {

Review Comment:
   I understand your point better!
   
   This is mostly a code architecture decision, based on what we think is cleaner in terms of implementation.
   I created the `IcebergRepartitionExec` to have a clear separation and identification (as a separate file and structure) to follow a common (and be integrated) DataFusion physical plan:
   ```
       -> InputExec
      -> ProjectionExec
      -> RepartitionExec
      -> SortExec
      -> WriteExec
   ```
   Therefore, we can easily identify the `IcebergReparationExec` implementation (file and DataFusion integration) and extend it, which can be used across different operations (merge, compaction...).
   
   To sum up:
   - I thought of having a dedicated file for the DataFusion plan to ensure we know that we apply an additional physical plan execution for Iceberg. It makes it clearer that there is a dedicated node in the plan that everyone could consider.
   - It's a safe encapsulation and composable by centralizing partitioning logic improves readability, maintainability, and evolution of the code.
   - Lastly, the logic could rapidly evolve, and we could introduce optimizations or more complex behavior, or implement custom DataFusion logic for performance tuning. I avoided this to keep it simple, but we 
   
   Of course, with your knowledge of the codebase, you know better how to determine the best place for this. 
   I'm thinking out loud, and we could move the logic to a more tailored implementation for now, where you think it makes more sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2314208788


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   I used a `HashSet` because I wanted to keep order of pa.
   If I use an iterator to collect into a `HashSet`, we could lose the order, resulting in incorrect partitioning (iterating in an arbitrary order).
   We could add `indexmap::IndexSet` as a dependency or use 'seen' to maintain the order with a hash set.
   
   In partitioning, order matters: partition identity columns must come before bucket columns. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2313515745


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,

Review Comment:
   Do we really need this parameter? I assume we can infer from table metadata.



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();
+    let hash_exprs: Vec<Arc<dyn datafusion::physical_expr::PhysicalExpr>> = names_iter
+        .filter(|name| seen.insert(*name))
+        .map(|name| {
+            let idx = input_schema
+                .index_of(name)
+                .map_err(|e| datafusion::error::DataFusionError::Plan(e.to_string()))?;
+            Ok(Arc::new(Column::new(name, idx))

Review Comment:
   Is this correct? I assume the project node should happen before repartition.



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(

Review Comment:
   ```suggestion
   pub(crate) fn repartition(
   ```
   I assume this will not  be used user?
   



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,

Review Comment:
   ```suggestion
       target_partitions: NonZeroUsize,
   ```



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {

Review Comment:
   I have concerns about considering sort order here. Iceberg's sort happens within a partition, not globally. For example, for following table:
   `partition by day(ts) order by bucket(id)`
   
   Grouping ids with same bucket value doesn't make the sort better. 



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   This is anti pattern, why not just make `names_iter` a `HashSet`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2324607217


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   > But the mapping from (columns) → (hash value) must be stable across plans, no?
   
   I still don't understand why this must be stable. To ensure correctness, we only need to ensure `partition_value_of(row1) = partition_value_of(row2)` if  `row1 == row2`. For be more exact, it even could be more relax like `row1(src_col1)==row2(src_col1) and row1(src_col2) == row2(src_col2)` for all source columns involved in   partition spec. Please not that the `partition_value_of` is calculated using iceberg's transform function, rather than datafusion's partition function.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2469849997


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,966 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering. This is used when the `_partition`
+///   column is present AND the partition spec has hash-friendly transforms (Identity/Bucket),
+///   or when source columns with these transforms are available.
+/// - **Round-robin partitioning** – Used for temporal transforms (Year, Month, Day, Hour),
+///   Truncate, or other transforms that don't provide good hash distribution.
+/// - **Mixed transforms** – Combines multiple transform types, using hash partitioning only
+///   when Identity or Bucket transforms are present, otherwise falling back to round-robin.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned. For partitioned tables,
+///   the input should include the `_partition` column (added via `project_with_partition`).
+/// * `table_metadata` – Iceberg table metadata containing partition spec.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+#[allow(dead_code)]
+pub(crate) fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {

Review Comment:
   Tell me what you think; I'm okay with removing this part of the integration of the other changes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2313751052


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(

Review Comment:
   True, it will be part of the `ExecutionPlan` building 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2314240257


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,

Review Comment:
   You are right, `schema` is part of the metadata with `current_schema()`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2297723455


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {

Review Comment:
   I think we just need to extend `PhysicalExpr`?



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {
+    /// Input execution plan
+    input: Arc<dyn ExecutionPlan>,
+    /// Iceberg table schema to determine partitioning strategy
+    table_schema: SchemaRef,
+    /// Iceberg table metadata to determine partitioning strategy
+    table_metadata: TableMetadataRef,
+    /// Target number of partitions for data distribution
+    target_partitions: usize,
+    /// Partitioning strategy
+    partitioning_strategy: Partitioning,
+    /// Plan properties for optimization
+    plan_properties: PlanProperties,
+}
+
+impl IcebergRepartitionExec {
+    /// Creates a new IcebergRepartitionExec with automatic partitioning strategy selection.
+    ///
+    /// This constructor analyzes the table's partition specification, sort order, and write
+    /// distribution mode to determine the optimal repartitioning strategy for insert operations.
+    ///
+    /// # Arguments
+    ///
+    /// * `input` - The input execution plan providing data to be repartitioned
+    /// * `table_schema` - The Iceberg table schema used to resolve column references
+    /// * `table_metadata` - The Iceberg table metadata containing partition spec, sort order,
+    ///   and table properties including write distribution mode
+    /// * `target_partitions` - Target number of partitions for parallel processing:
+    ///   - `0`: Auto-detect from DataFusion's SessionConfig target_partitions (recommended)
+    ///   - `> 0`: Use explicit partition count for specific performance tuning
+    ///
+    /// # Returns
+    ///
+    /// A configured repartition execution plan that will apply the optimal partitioning
+    /// strategy during execution, or pass through unchanged data if no repartitioning
+    /// is needed.
+    ///
+    /// # Example
+    ///
+    /// ```ignore
+    /// let repartition_exec = IcebergRepartitionExec::new(
+    ///     input_plan,
+    ///     table.schema_ref(),
+    ///     table.metadata_ref(),
+    ///     state.config().target_partitions(),
+    /// )?;
+    /// ```
+    pub fn new(
+        input: Arc<dyn ExecutionPlan>,
+        table_schema: SchemaRef,
+        table_metadata: TableMetadataRef,
+        target_partitions: usize,
+    ) -> DFResult<Self> {
+        if target_partitions == 0 {
+            return Err(datafusion::error::DataFusionError::Plan(
+                "IcebergRepartitionExec requires target_partitions > 0".to_string(),
+            ));
+        }
+
+        let partitioning_strategy = Self::determine_partitioning_strategy(
+            &input,
+            &table_schema,
+            &table_metadata,
+            target_partitions,
+        )?;
+
+        let plan_properties = Self::compute_properties(&input, &partitioning_strategy)?;
+
+        Ok(Self {
+            input,
+            table_schema,
+            table_metadata,
+            target_partitions,
+            partitioning_strategy,
+            plan_properties,
+        })
+    }
+
+    /// Computes the plan properties based on the table partitioning strategy
+    /// Selects the partitioning strategy based on the table partitioning strategy
+    fn compute_properties(
+        input: &Arc<dyn ExecutionPlan>,
+        partitioning_strategy: &Partitioning,
+    ) -> DFResult<PlanProperties> {
+        let schema = input.schema();
+        let equivalence_properties = EquivalenceProperties::new(schema);
+
+        Ok(PlanProperties::new(
+            equivalence_properties,
+            partitioning_strategy.clone(),
+            EmissionType::Incremental,
+            Boundedness::Bounded,
+        ))
+    }
+
+    /// Determines the optimal partitioning strategy based on table metadata and distribution mode.
+    ///
+    /// This function analyzes the table's partition specification, sort order, and write distribution
+    /// mode to select the most appropriate DataFusion partitioning strategy for insert operations.
+    ///
+    /// ## Distribution Mode Logic
+    ///
+    /// The strategy is determined by the table's `write.distribution-mode` property:
+    ///
+    /// - **`hash` (default)**: Uses hash partitioning on:
+    ///   1. Identity partition columns (e.g., `PARTITIONED BY (year, month)`)
+    ///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+    ///   3. Bucket columns from sort order
+    ///   
+    ///   This ensures data co-location within partitions and buckets for optimal file clustering.
+    ///
+    /// - **`range`**: Approximates range distribution by hashing on sort order columns.
+    ///   Since DataFusion lacks native range exchange, this provides the closest alternative
+    ///   while maintaining some ordering characteristics.
+    ///
+    /// - **`none` or other**: Falls back to round-robin distribution for balanced load.
+    ///
+    /// ## Column Priority and Deduplication
+    ///
+    /// When multiple column sources are available, they are combined in this order:
+    /// 1. Partition identity columns (highest priority)
+    /// 2. Bucket columns from partition spec  
+    /// 3. Bucket columns from sort order
+    /// 4. Sort order columns (for range mode)
+    ///
+    /// Duplicate columns are automatically removed while preserving the priority order.
+    ///
+    /// ## Fallback Behavior
+    ///
+    /// If no suitable hash columns are found (e.g., unpartitioned, non-bucketed table),
+    /// falls back to round-robin batch partitioning for even load distribution.
+    fn determine_partitioning_strategy(

Review Comment:
   This is interesting to see. At first I just thought two cases:
   1. If it's partitioned table, we should just hash partition.
   2. If it's not partitioned, we should just use round robin partition.
   
   However, this reminds me another case: range only partition, e.g. we only has partitions like date, time. I think in this case we should also use round robin partition since in this case most data are focused in several partitions.
   
   Also I don't think we should take  into account `write.distribution-mode` for now. The example  you use are for spark, but not  applicable for datafusion.



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {

Review Comment:
   There already exists an [RepartitionExec](https://github.com/apache/datafusion/blob/8b03e5e0050c673393f7e8d1c6455cf9ff7e2570/datafusion/physical-plan/src/repartition/mod.rs#L831), why we need to create a new one? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2465153774


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,798 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification and sort order.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering.
+/// - **Round-robin partitioning** – Used for range-only or temporal transforms that don’t
+///   provide good distribution.
+/// - **Mixed transforms** – Combines range and identity/bucket transforms, using hash
+///   partitioning on identity/bucket columns.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned (already projected to match table schema).
+/// * `table_metadata` – Iceberg table metadata containing partition spec and sort order.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(a_exprs: &[Arc<dyn PhysicalExpr>], b_exprs: &[Arc<dyn PhysicalExpr>]) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs.iter().zip(b_exprs.iter()).all(|(a, b)| {
+        if let (Some(a_col), Some(b_col)) = (
+            a.as_any().downcast_ref::<Column>(),
+            b.as_any().downcast_ref::<Column>(),
+        ) {
+            a_col.name() == b_col.name() && a_col.index() == b_col.index()
+        } else {
+            std::ptr::eq(a.as_ref(), b.as_ref())
+        }
+    })
+}
+
+/// Determine the optimal partitioning strategy based on table metadata.
+///
+/// Analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy
+///
+/// - **Hash partitioning using `_partition` column**: Used when the input includes a
+///   projected `_partition` column. Ensures data is distributed based on actual partition values.
+///
+/// - **Hash partitioning using source columns**: Applied when identity or bucket transforms
+///   provide good distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order  
+///   Ensures co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for unpartitioned, range-only, or non-bucketed tables
+///   where hash partitioning is not feasible.
+///
+/// ## Column Priority
+///
+/// Columns are combined in the following order, with duplicates removed:
+/// 1. `_partition` column (highest priority, if present)
+/// 2. Identity partition columns
+/// 3. Bucket columns from partition spec
+/// 4. Bucket columns from sort order
+///
+/// ## Fallback
+///
+/// If no suitable hash columns are found, falls back to round-robin batch partitioning
+/// to ensure even load distribution across partitions.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_metadata: &TableMetadata,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let table_schema = table_metadata.current_schema();
+    let input_schema = input.schema();
+
+    match (
+        !partition_spec.is_unpartitioned(),
+        input_schema.index_of(PARTITION_VALUES_COLUMN),
+    ) {
+        (true, Ok(partition_col_idx)) => {
+            let partition_field = input_schema.field(partition_col_idx);
+            if partition_field.name() != PARTITION_VALUES_COLUMN {
+                return Err(DataFusionError::Plan(format!(
+                    "Expected {} column at index {}, but found '{}'",
+                    PARTITION_VALUES_COLUMN,
+                    partition_col_idx,
+                    partition_field.name()
+                )));
+            }
+
+            let partition_expr = Arc::new(Column::new(PARTITION_VALUES_COLUMN, partition_col_idx))
+                as Arc<dyn PhysicalExpr>;
+            return Ok(Partitioning::Hash(

Review Comment:
   Right, I reworked the strategy by including the _partition column, but I have forgotten one case here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2469691399


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,966 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering. This is used when the `_partition`
+///   column is present AND the partition spec has hash-friendly transforms (Identity/Bucket),
+///   or when source columns with these transforms are available.
+/// - **Round-robin partitioning** – Used for temporal transforms (Year, Month, Day, Hour),
+///   Truncate, or other transforms that don't provide good hash distribution.
+/// - **Mixed transforms** – Combines multiple transform types, using hash partitioning only
+///   when Identity or Bucket transforms are present, otherwise falling back to round-robin.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned. For partitioned tables,
+///   the input should include the `_partition` column (added via `project_with_partition`).
+/// * `table_metadata` – Iceberg table metadata containing partition spec.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+#[allow(dead_code)]
+pub(crate) fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(a_exprs: &[Arc<dyn PhysicalExpr>], b_exprs: &[Arc<dyn PhysicalExpr>]) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs.iter().zip(b_exprs.iter()).all(|(a, b)| {
+        if let (Some(a_col), Some(b_col)) = (
+            a.as_any().downcast_ref::<Column>(),
+            b.as_any().downcast_ref::<Column>(),
+        ) {
+            a_col.name() == b_col.name() && a_col.index() == b_col.index()
+        } else {
+            std::ptr::eq(a.as_ref(), b.as_ref())
+        }
+    })
+}
+
+/// Determine the optimal partitioning strategy based on table metadata.
+///
+/// Analyzes the table's partition specification to select the most appropriate
+/// DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy
+///
+/// - **Hash partitioning using `_partition` column**: Used when the input includes a
+///   projected `_partition` column AND the partition spec contains Identity or Bucket transforms.
+///   Ensures data is distributed based on actual partition values with good distribution.
+///
+/// - **Hash partitioning using source columns**: Applied when identity or bucket transforms
+///   provide good distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///
+///   Ensures co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for unpartitioned tables, or when partition transforms
+///   don't provide good hash distribution (e.g., Year, Month, Day, Hour, Truncate transforms).
+///   Ensures even load distribution across partitions.
+///
+/// ## Column Priority
+///
+/// Columns are combined in the following order, with duplicates removed:
+/// 1. `_partition` column (highest priority, if present)
+/// 2. Identity partition columns from partition spec
+/// 3. Bucket columns from partition spec
+///
+/// ## Fallback
+///
+/// If no suitable hash columns are found, falls back to round-robin batch partitioning
+/// to ensure even load distribution across partitions.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_metadata: &TableMetadata,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let table_schema = table_metadata.current_schema();
+    let input_schema = input.schema();
+    let target_partition_count = target_partitions.get();
+
+    // Check if partition spec has transforms suitable for hash partitioning
+    let has_hash_friendly_transforms = partition_spec
+        .fields()
+        .iter()
+        .any(|pf| matches!(pf.transform, Transform::Identity | Transform::Bucket(_)));
+
+    let partition_col_result = input_schema.index_of(PARTITION_VALUES_COLUMN);
+    let is_partitioned_table = !partition_spec.is_unpartitioned();
+
+    match (is_partitioned_table, partition_col_result) {
+        // Case 1: Partitioned table with _partition column present
+        (true, Ok(partition_col_idx)) => {
+            let partition_field = input_schema.field(partition_col_idx);
+            if partition_field.name() != PARTITION_VALUES_COLUMN {

Review Comment:
   You're right! 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2469734510


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,966 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering. This is used when the `_partition`
+///   column is present AND the partition spec has hash-friendly transforms (Identity/Bucket),
+///   or when source columns with these transforms are available.
+/// - **Round-robin partitioning** – Used for temporal transforms (Year, Month, Day, Hour),
+///   Truncate, or other transforms that don't provide good hash distribution.
+/// - **Mixed transforms** – Combines multiple transform types, using hash partitioning only
+///   when Identity or Bucket transforms are present, otherwise falling back to round-robin.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned. For partitioned tables,
+///   the input should include the `_partition` column (added via `project_with_partition`).
+/// * `table_metadata` – Iceberg table metadata containing partition spec.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+#[allow(dead_code)]
+pub(crate) fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(a_exprs: &[Arc<dyn PhysicalExpr>], b_exprs: &[Arc<dyn PhysicalExpr>]) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs.iter().zip(b_exprs.iter()).all(|(a, b)| {
+        if let (Some(a_col), Some(b_col)) = (
+            a.as_any().downcast_ref::<Column>(),
+            b.as_any().downcast_ref::<Column>(),
+        ) {
+            a_col.name() == b_col.name() && a_col.index() == b_col.index()
+        } else {
+            std::ptr::eq(a.as_ref(), b.as_ref())
+        }
+    })
+}
+
+/// Determine the optimal partitioning strategy based on table metadata.
+///
+/// Analyzes the table's partition specification to select the most appropriate
+/// DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy
+///
+/// - **Hash partitioning using `_partition` column**: Used when the input includes a
+///   projected `_partition` column AND the partition spec contains Identity or Bucket transforms.
+///   Ensures data is distributed based on actual partition values with good distribution.
+///
+/// - **Hash partitioning using source columns**: Applied when identity or bucket transforms
+///   provide good distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///
+///   Ensures co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for unpartitioned tables, or when partition transforms
+///   don't provide good hash distribution (e.g., Year, Month, Day, Hour, Truncate transforms).
+///   Ensures even load distribution across partitions.
+///
+/// ## Column Priority
+///
+/// Columns are combined in the following order, with duplicates removed:
+/// 1. `_partition` column (highest priority, if present)
+/// 2. Identity partition columns from partition spec
+/// 3. Bucket columns from partition spec
+///
+/// ## Fallback
+///
+/// If no suitable hash columns are found, falls back to round-robin batch partitioning
+/// to ensure even load distribution across partitions.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_metadata: &TableMetadata,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let table_schema = table_metadata.current_schema();
+    let input_schema = input.schema();
+    let target_partition_count = target_partitions.get();
+
+    // Check if partition spec has transforms suitable for hash partitioning
+    let has_hash_friendly_transforms = partition_spec
+        .fields()
+        .iter()
+        .any(|pf| matches!(pf.transform, Transform::Identity | Transform::Bucket(_)));
+
+    let partition_col_result = input_schema.index_of(PARTITION_VALUES_COLUMN);
+    let is_partitioned_table = !partition_spec.is_unpartitioned();
+
+    match (is_partitioned_table, partition_col_result) {
+        // Case 1: Partitioned table with _partition column present
+        (true, Ok(partition_col_idx)) => {
+            let partition_field = input_schema.field(partition_col_idx);
+            if partition_field.name() != PARTITION_VALUES_COLUMN {
+                return Err(DataFusionError::Plan(format!(
+                    "Expected {} column at index {}, but found '{}'",
+                    PARTITION_VALUES_COLUMN,
+                    partition_col_idx,
+                    partition_field.name()
+                )));
+            }
+
+            let partition_expr = Arc::new(Column::new(PARTITION_VALUES_COLUMN, partition_col_idx))
+                as Arc<dyn PhysicalExpr>;
+
+            return if has_hash_friendly_transforms {
+                Ok(Partitioning::Hash(
+                    vec![partition_expr],
+                    target_partition_count,
+                ))
+            } else {
+                Ok(Partitioning::RoundRobinBatch(target_partition_count))
+            };
+        }
+
+        // Case 2: Partitioned table missing _partition column (warning)
+        (true, Err(_)) => {
+            tracing::warn!(
+                "Partitioned table input missing {} column. \
+                 Consider adding partition projection before repartitioning.",
+                PARTITION_VALUES_COLUMN
+            );
+        }
+
+        // Case 3: Unpartitioned table with _partition column
+        (false, Ok(_)) => {
+            tracing::warn!(
+                "Input contains {} column but table is unpartitioned. \
+                 This may indicate unnecessary projection.",
+                PARTITION_VALUES_COLUMN
+            );
+        }
+
+        // Case 4: Unpartitioned table without _partition column
+        (false, Err(_)) => {
+            // Nothing to do - fall through to source column analysis
+        }
+    }
+
+    let hash_column_names: Vec<&str> = partition_spec

Review Comment:
   Since the physical plan is generated by `IcebergTableProvider`, which always calls `project_with_partition()` for partitioned tables... we can guarantee that the `_partition` column will be present... so it is unnecessary if we stick to the DataFusion plan creation! Let me rework this. I will add the need for the column `_partition` to be available.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2465061090


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,798 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification and sort order.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering.
+/// - **Round-robin partitioning** – Used for range-only or temporal transforms that don’t
+///   provide good distribution.
+/// - **Mixed transforms** – Combines range and identity/bucket transforms, using hash
+///   partitioning on identity/bucket columns.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned (already projected to match table schema).
+/// * `table_metadata` – Iceberg table metadata containing partition spec and sort order.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(a_exprs: &[Arc<dyn PhysicalExpr>], b_exprs: &[Arc<dyn PhysicalExpr>]) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs.iter().zip(b_exprs.iter()).all(|(a, b)| {
+        if let (Some(a_col), Some(b_col)) = (
+            a.as_any().downcast_ref::<Column>(),
+            b.as_any().downcast_ref::<Column>(),
+        ) {
+            a_col.name() == b_col.name() && a_col.index() == b_col.index()
+        } else {
+            std::ptr::eq(a.as_ref(), b.as_ref())
+        }
+    })
+}
+
+/// Determine the optimal partitioning strategy based on table metadata.
+///
+/// Analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy
+///
+/// - **Hash partitioning using `_partition` column**: Used when the input includes a
+///   projected `_partition` column. Ensures data is distributed based on actual partition values.
+///
+/// - **Hash partitioning using source columns**: Applied when identity or bucket transforms
+///   provide good distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order  
+///   Ensures co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for unpartitioned, range-only, or non-bucketed tables
+///   where hash partitioning is not feasible.
+///
+/// ## Column Priority
+///
+/// Columns are combined in the following order, with duplicates removed:
+/// 1. `_partition` column (highest priority, if present)
+/// 2. Identity partition columns
+/// 3. Bucket columns from partition spec
+/// 4. Bucket columns from sort order
+///
+/// ## Fallback
+///
+/// If no suitable hash columns are found, falls back to round-robin batch partitioning
+/// to ensure even load distribution across partitions.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_metadata: &TableMetadata,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let table_schema = table_metadata.current_schema();
+    let input_schema = input.schema();
+
+    match (
+        !partition_spec.is_unpartitioned(),
+        input_schema.index_of(PARTITION_VALUES_COLUMN),
+    ) {
+        (true, Ok(partition_col_idx)) => {
+            let partition_field = input_schema.field(partition_col_idx);
+            if partition_field.name() != PARTITION_VALUES_COLUMN {
+                return Err(DataFusionError::Plan(format!(
+                    "Expected {} column at index {}, but found '{}'",
+                    PARTITION_VALUES_COLUMN,
+                    partition_col_idx,
+                    partition_field.name()
+                )));
+            }
+
+            let partition_expr = Arc::new(Column::new(PARTITION_VALUES_COLUMN, partition_col_idx))
+                as Arc<dyn PhysicalExpr>;
+            return Ok(Partitioning::Hash(

Review Comment:
   This is incorrect? We should move the logic to determine hash or range here?



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(

Review Comment:
   This is not resolved?



##########
crates/iceberg/src/table.rs:
##########
@@ -235,6 +235,11 @@ impl Table {
         self.readonly
     }
 
+    /// Returns the current schema as a shared reference.
+    pub fn schema_ref(&self) -> SchemaRef {

Review Comment:
   ```suggestion
       pub fn current_schema_ref(&self) -> SchemaRef {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2468875080


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,966 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering. This is used when the `_partition`
+///   column is present AND the partition spec has hash-friendly transforms (Identity/Bucket),
+///   or when source columns with these transforms are available.
+/// - **Round-robin partitioning** – Used for temporal transforms (Year, Month, Day, Hour),
+///   Truncate, or other transforms that don't provide good hash distribution.
+/// - **Mixed transforms** – Combines multiple transform types, using hash partitioning only
+///   when Identity or Bucket transforms are present, otherwise falling back to round-robin.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned. For partitioned tables,
+///   the input should include the `_partition` column (added via `project_with_partition`).
+/// * `table_metadata` – Iceberg table metadata containing partition spec.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+#[allow(dead_code)]
+pub(crate) fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {

Review Comment:
   I'm confused why we need this part. For partitioned table, the `input` is expected to be a `ProjectExec`, and we should always add the partitioning. For unpartitioned table, we should simply return `RoundRobinBatch`



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,966 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering. This is used when the `_partition`
+///   column is present AND the partition spec has hash-friendly transforms (Identity/Bucket),
+///   or when source columns with these transforms are available.
+/// - **Round-robin partitioning** – Used for temporal transforms (Year, Month, Day, Hour),
+///   Truncate, or other transforms that don't provide good hash distribution.
+/// - **Mixed transforms** – Combines multiple transform types, using hash partitioning only
+///   when Identity or Bucket transforms are present, otherwise falling back to round-robin.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned. For partitioned tables,
+///   the input should include the `_partition` column (added via `project_with_partition`).
+/// * `table_metadata` – Iceberg table metadata containing partition spec.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+#[allow(dead_code)]
+pub(crate) fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(a_exprs: &[Arc<dyn PhysicalExpr>], b_exprs: &[Arc<dyn PhysicalExpr>]) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs.iter().zip(b_exprs.iter()).all(|(a, b)| {
+        if let (Some(a_col), Some(b_col)) = (
+            a.as_any().downcast_ref::<Column>(),
+            b.as_any().downcast_ref::<Column>(),
+        ) {
+            a_col.name() == b_col.name() && a_col.index() == b_col.index()
+        } else {
+            std::ptr::eq(a.as_ref(), b.as_ref())
+        }
+    })
+}
+
+/// Determine the optimal partitioning strategy based on table metadata.
+///
+/// Analyzes the table's partition specification to select the most appropriate
+/// DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy
+///
+/// - **Hash partitioning using `_partition` column**: Used when the input includes a
+///   projected `_partition` column AND the partition spec contains Identity or Bucket transforms.
+///   Ensures data is distributed based on actual partition values with good distribution.
+///
+/// - **Hash partitioning using source columns**: Applied when identity or bucket transforms
+///   provide good distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///
+///   Ensures co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for unpartitioned tables, or when partition transforms
+///   don't provide good hash distribution (e.g., Year, Month, Day, Hour, Truncate transforms).
+///   Ensures even load distribution across partitions.
+///
+/// ## Column Priority
+///
+/// Columns are combined in the following order, with duplicates removed:
+/// 1. `_partition` column (highest priority, if present)
+/// 2. Identity partition columns from partition spec
+/// 3. Bucket columns from partition spec
+///
+/// ## Fallback
+///
+/// If no suitable hash columns are found, falls back to round-robin batch partitioning
+/// to ensure even load distribution across partitions.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_metadata: &TableMetadata,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let table_schema = table_metadata.current_schema();
+    let input_schema = input.schema();
+    let target_partition_count = target_partitions.get();
+
+    // Check if partition spec has transforms suitable for hash partitioning
+    let has_hash_friendly_transforms = partition_spec
+        .fields()
+        .iter()
+        .any(|pf| matches!(pf.transform, Transform::Identity | Transform::Bucket(_)));
+
+    let partition_col_result = input_schema.index_of(PARTITION_VALUES_COLUMN);
+    let is_partitioned_table = !partition_spec.is_unpartitioned();
+
+    match (is_partitioned_table, partition_col_result) {
+        // Case 1: Partitioned table with _partition column present
+        (true, Ok(partition_col_idx)) => {
+            let partition_field = input_schema.field(partition_col_idx);
+            if partition_field.name() != PARTITION_VALUES_COLUMN {

Review Comment:
   nit: This seems unnecessary?



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,966 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering. This is used when the `_partition`
+///   column is present AND the partition spec has hash-friendly transforms (Identity/Bucket),
+///   or when source columns with these transforms are available.
+/// - **Round-robin partitioning** – Used for temporal transforms (Year, Month, Day, Hour),
+///   Truncate, or other transforms that don't provide good hash distribution.
+/// - **Mixed transforms** – Combines multiple transform types, using hash partitioning only
+///   when Identity or Bucket transforms are present, otherwise falling back to round-robin.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned. For partitioned tables,
+///   the input should include the `_partition` column (added via `project_with_partition`).
+/// * `table_metadata` – Iceberg table metadata containing partition spec.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+#[allow(dead_code)]
+pub(crate) fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(a_exprs: &[Arc<dyn PhysicalExpr>], b_exprs: &[Arc<dyn PhysicalExpr>]) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs.iter().zip(b_exprs.iter()).all(|(a, b)| {
+        if let (Some(a_col), Some(b_col)) = (
+            a.as_any().downcast_ref::<Column>(),
+            b.as_any().downcast_ref::<Column>(),
+        ) {
+            a_col.name() == b_col.name() && a_col.index() == b_col.index()
+        } else {
+            std::ptr::eq(a.as_ref(), b.as_ref())
+        }
+    })
+}
+
+/// Determine the optimal partitioning strategy based on table metadata.
+///
+/// Analyzes the table's partition specification to select the most appropriate
+/// DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy
+///
+/// - **Hash partitioning using `_partition` column**: Used when the input includes a
+///   projected `_partition` column AND the partition spec contains Identity or Bucket transforms.
+///   Ensures data is distributed based on actual partition values with good distribution.
+///
+/// - **Hash partitioning using source columns**: Applied when identity or bucket transforms
+///   provide good distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///
+///   Ensures co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for unpartitioned tables, or when partition transforms
+///   don't provide good hash distribution (e.g., Year, Month, Day, Hour, Truncate transforms).
+///   Ensures even load distribution across partitions.
+///
+/// ## Column Priority
+///
+/// Columns are combined in the following order, with duplicates removed:
+/// 1. `_partition` column (highest priority, if present)
+/// 2. Identity partition columns from partition spec
+/// 3. Bucket columns from partition spec
+///
+/// ## Fallback
+///
+/// If no suitable hash columns are found, falls back to round-robin batch partitioning
+/// to ensure even load distribution across partitions.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_metadata: &TableMetadata,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let table_schema = table_metadata.current_schema();
+    let input_schema = input.schema();
+    let target_partition_count = target_partitions.get();
+
+    // Check if partition spec has transforms suitable for hash partitioning
+    let has_hash_friendly_transforms = partition_spec
+        .fields()
+        .iter()
+        .any(|pf| matches!(pf.transform, Transform::Identity | Transform::Bucket(_)));
+
+    let partition_col_result = input_schema.index_of(PARTITION_VALUES_COLUMN);
+    let is_partitioned_table = !partition_spec.is_unpartitioned();
+
+    match (is_partitioned_table, partition_col_result) {
+        // Case 1: Partitioned table with _partition column present
+        (true, Ok(partition_col_idx)) => {
+            let partition_field = input_schema.field(partition_col_idx);
+            if partition_field.name() != PARTITION_VALUES_COLUMN {
+                return Err(DataFusionError::Plan(format!(
+                    "Expected {} column at index {}, but found '{}'",
+                    PARTITION_VALUES_COLUMN,
+                    partition_col_idx,
+                    partition_field.name()
+                )));
+            }
+
+            let partition_expr = Arc::new(Column::new(PARTITION_VALUES_COLUMN, partition_col_idx))
+                as Arc<dyn PhysicalExpr>;
+
+            return if has_hash_friendly_transforms {
+                Ok(Partitioning::Hash(
+                    vec![partition_expr],
+                    target_partition_count,
+                ))
+            } else {
+                Ok(Partitioning::RoundRobinBatch(target_partition_count))
+            };
+        }
+
+        // Case 2: Partitioned table missing _partition column (warning)
+        (true, Err(_)) => {
+            tracing::warn!(
+                "Partitioned table input missing {} column. \
+                 Consider adding partition projection before repartitioning.",
+                PARTITION_VALUES_COLUMN
+            );
+        }
+
+        // Case 3: Unpartitioned table with _partition column
+        (false, Ok(_)) => {
+            tracing::warn!(
+                "Input contains {} column but table is unpartitioned. \
+                 This may indicate unnecessary projection.",
+                PARTITION_VALUES_COLUMN
+            );
+        }
+
+        // Case 4: Unpartitioned table without _partition column
+        (false, Err(_)) => {
+            // Nothing to do - fall through to source column analysis
+        }
+    }
+
+    let hash_column_names: Vec<&str> = partition_spec

Review Comment:
   Why we need this part? The physical plan is generated by `IcebergTableProvider`, so we could ensure that the `_partition` column will be inserted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2469852646


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();
+    let hash_exprs: Vec<Arc<dyn datafusion::physical_expr::PhysicalExpr>> = names_iter
+        .filter(|name| seen.insert(*name))
+        .map(|name| {
+            let idx = input_schema
+                .index_of(name)
+                .map_err(|e| datafusion::error::DataFusionError::Plan(e.to_string()))?;
+            Ok(Arc::new(Column::new(name, idx))

Review Comment:
   Making clear to leverage `project_with_partitions()` first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2306812500


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {

Review Comment:
   I'm fine with a dedicated file for repartition logic. My only concern is maintaining a dedicated executor for repartitions. The interface of this repartition module coulde be as simple as
   ```
   pub fn repartition(input: Arc<dyn ExecutionPlan>, ...) -> Result<Arc<dyn ExecutionPlan> {
   }
   ```
   
   This interface is still extensible even if you plan to add new repartition logic in future. I agree that there are some potentital implemenation to replace `RepartitionExec` in datafusion, but we could add it when we have concrete example. Let's keep things small and clean at first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2315641228


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   No worries!
   Let's take an example to see if there's a problem with data consistency if we are iterating without keeping order across two plans.
   
   Data:
   ```
   Row 1: a=5, b=10
   Row 2: a=5, b=10  
   Row 3: a=7, b=3
   ```
   1. HashPartition(vec![a, b], 4)
   ```
   Row 1: hash(5, 10) = 1234 → 1234 % 4 = 2 (goes to partition 2)
   Row 2: hash(5, 10) = 1234 → 1234 % 4 = 2 (goes to partition 2)
   Row 3: hash(7, 3)  = 5678 → 5678 % 4 = 2 (goes to partition 2)
   ```
   2. HashPartition(vec![b, a], 4)
   ```
   Row 1: hash(10, 5) = 8765 → 8765 % 4 = 1 (goes to partition 1)
   Row 2: hash(10, 5) = 8765 → 8765 % 4 = 1 (goes to partition 1)
   Row 3: hash(3, 7)  = 4321 → 4321 % 4 = 1 (goes to partition 1)
   ```
   
   Same data, different distributions:
   Plan 1: Rows 1,2,3 all go to partition 2
   Plan 2: Rows 1,2,3 all go to partition 1
   
   If we are iterating in an arbitrary order on hash partitioning, we could have different plans leading to not ending in the right partition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#issuecomment-3471929364

   > Thanks @fvaleye for this pr, and sorry for late reply. I think we are almost done, the only left things to do is removing redundant check.
   
   No worries @liurenjie1024! 
   Thanks, let's finish this work! I will make the latest required changes! 🚀 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#issuecomment-3471998327

   > Thanks @fvaleye for this pr!
   
   I noticed that the constant `PARTITION_VALUES_COLUMN` was refactored and renamed to `PROJECTED_PARTITION_VALUE_COLUMN` in the Iceberg crate. Let me fix this, and we are good!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#issuecomment-3472043763

   > > Thanks @fvaleye for this pr!
   > 
   > I noticed that the constant `PARTITION_VALUES_COLUMN` was refactored and renamed to `PROJECTED_PARTITION_VALUE_COLUMN` in the Iceberg crate. Let me fix this, and we are good!
   
   All good!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 merged PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2469734510


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,966 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering. This is used when the `_partition`
+///   column is present AND the partition spec has hash-friendly transforms (Identity/Bucket),
+///   or when source columns with these transforms are available.
+/// - **Round-robin partitioning** – Used for temporal transforms (Year, Month, Day, Hour),
+///   Truncate, or other transforms that don't provide good hash distribution.
+/// - **Mixed transforms** – Combines multiple transform types, using hash partitioning only
+///   when Identity or Bucket transforms are present, otherwise falling back to round-robin.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned. For partitioned tables,
+///   the input should include the `_partition` column (added via `project_with_partition`).
+/// * `table_metadata` – Iceberg table metadata containing partition spec.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+#[allow(dead_code)]
+pub(crate) fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(a_exprs: &[Arc<dyn PhysicalExpr>], b_exprs: &[Arc<dyn PhysicalExpr>]) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs.iter().zip(b_exprs.iter()).all(|(a, b)| {
+        if let (Some(a_col), Some(b_col)) = (
+            a.as_any().downcast_ref::<Column>(),
+            b.as_any().downcast_ref::<Column>(),
+        ) {
+            a_col.name() == b_col.name() && a_col.index() == b_col.index()
+        } else {
+            std::ptr::eq(a.as_ref(), b.as_ref())
+        }
+    })
+}
+
+/// Determine the optimal partitioning strategy based on table metadata.
+///
+/// Analyzes the table's partition specification to select the most appropriate
+/// DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy
+///
+/// - **Hash partitioning using `_partition` column**: Used when the input includes a
+///   projected `_partition` column AND the partition spec contains Identity or Bucket transforms.
+///   Ensures data is distributed based on actual partition values with good distribution.
+///
+/// - **Hash partitioning using source columns**: Applied when identity or bucket transforms
+///   provide good distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///
+///   Ensures co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for unpartitioned tables, or when partition transforms
+///   don't provide good hash distribution (e.g., Year, Month, Day, Hour, Truncate transforms).
+///   Ensures even load distribution across partitions.
+///
+/// ## Column Priority
+///
+/// Columns are combined in the following order, with duplicates removed:
+/// 1. `_partition` column (highest priority, if present)
+/// 2. Identity partition columns from partition spec
+/// 3. Bucket columns from partition spec
+///
+/// ## Fallback
+///
+/// If no suitable hash columns are found, falls back to round-robin batch partitioning
+/// to ensure even load distribution across partitions.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_metadata: &TableMetadata,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let table_schema = table_metadata.current_schema();
+    let input_schema = input.schema();
+    let target_partition_count = target_partitions.get();
+
+    // Check if partition spec has transforms suitable for hash partitioning
+    let has_hash_friendly_transforms = partition_spec
+        .fields()
+        .iter()
+        .any(|pf| matches!(pf.transform, Transform::Identity | Transform::Bucket(_)));
+
+    let partition_col_result = input_schema.index_of(PARTITION_VALUES_COLUMN);
+    let is_partitioned_table = !partition_spec.is_unpartitioned();
+
+    match (is_partitioned_table, partition_col_result) {
+        // Case 1: Partitioned table with _partition column present
+        (true, Ok(partition_col_idx)) => {
+            let partition_field = input_schema.field(partition_col_idx);
+            if partition_field.name() != PARTITION_VALUES_COLUMN {
+                return Err(DataFusionError::Plan(format!(
+                    "Expected {} column at index {}, but found '{}'",
+                    PARTITION_VALUES_COLUMN,
+                    partition_col_idx,
+                    partition_field.name()
+                )));
+            }
+
+            let partition_expr = Arc::new(Column::new(PARTITION_VALUES_COLUMN, partition_col_idx))
+                as Arc<dyn PhysicalExpr>;
+
+            return if has_hash_friendly_transforms {
+                Ok(Partitioning::Hash(
+                    vec![partition_expr],
+                    target_partition_count,
+                ))
+            } else {
+                Ok(Partitioning::RoundRobinBatch(target_partition_count))
+            };
+        }
+
+        // Case 2: Partitioned table missing _partition column (warning)
+        (true, Err(_)) => {
+            tracing::warn!(
+                "Partitioned table input missing {} column. \
+                 Consider adding partition projection before repartitioning.",
+                PARTITION_VALUES_COLUMN
+            );
+        }
+
+        // Case 3: Unpartitioned table with _partition column
+        (false, Ok(_)) => {
+            tracing::warn!(
+                "Input contains {} column but table is unpartitioned. \
+                 This may indicate unnecessary projection.",
+                PARTITION_VALUES_COLUMN
+            );
+        }
+
+        // Case 4: Unpartitioned table without _partition column
+        (false, Err(_)) => {
+            // Nothing to do - fall through to source column analysis
+        }
+    }
+
+    let hash_column_names: Vec<&str> = partition_spec

Review Comment:
   Since the physical plan is generated by `IcebergTableProvider`, which always calls `project_with_partition()` for partitioned tables... we can guarantee that the `_partition` column will be present... so it is unnecessary if we stick to the DataFusion plan creation! Let me rework this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2297894049


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {
+    /// Input execution plan
+    input: Arc<dyn ExecutionPlan>,
+    /// Iceberg table schema to determine partitioning strategy
+    table_schema: SchemaRef,
+    /// Iceberg table metadata to determine partitioning strategy
+    table_metadata: TableMetadataRef,
+    /// Target number of partitions for data distribution
+    target_partitions: usize,
+    /// Partitioning strategy
+    partitioning_strategy: Partitioning,
+    /// Plan properties for optimization
+    plan_properties: PlanProperties,
+}
+
+impl IcebergRepartitionExec {
+    /// Creates a new IcebergRepartitionExec with automatic partitioning strategy selection.
+    ///
+    /// This constructor analyzes the table's partition specification, sort order, and write
+    /// distribution mode to determine the optimal repartitioning strategy for insert operations.
+    ///
+    /// # Arguments
+    ///
+    /// * `input` - The input execution plan providing data to be repartitioned
+    /// * `table_schema` - The Iceberg table schema used to resolve column references
+    /// * `table_metadata` - The Iceberg table metadata containing partition spec, sort order,
+    ///   and table properties including write distribution mode
+    /// * `target_partitions` - Target number of partitions for parallel processing:
+    ///   - `0`: Auto-detect from DataFusion's SessionConfig target_partitions (recommended)
+    ///   - `> 0`: Use explicit partition count for specific performance tuning
+    ///
+    /// # Returns
+    ///
+    /// A configured repartition execution plan that will apply the optimal partitioning
+    /// strategy during execution, or pass through unchanged data if no repartitioning
+    /// is needed.
+    ///
+    /// # Example
+    ///
+    /// ```ignore
+    /// let repartition_exec = IcebergRepartitionExec::new(
+    ///     input_plan,
+    ///     table.schema_ref(),
+    ///     table.metadata_ref(),
+    ///     state.config().target_partitions(),
+    /// )?;
+    /// ```
+    pub fn new(
+        input: Arc<dyn ExecutionPlan>,
+        table_schema: SchemaRef,
+        table_metadata: TableMetadataRef,
+        target_partitions: usize,
+    ) -> DFResult<Self> {
+        if target_partitions == 0 {
+            return Err(datafusion::error::DataFusionError::Plan(
+                "IcebergRepartitionExec requires target_partitions > 0".to_string(),
+            ));
+        }
+
+        let partitioning_strategy = Self::determine_partitioning_strategy(
+            &input,
+            &table_schema,
+            &table_metadata,
+            target_partitions,
+        )?;
+
+        let plan_properties = Self::compute_properties(&input, &partitioning_strategy)?;
+
+        Ok(Self {
+            input,
+            table_schema,
+            table_metadata,
+            target_partitions,
+            partitioning_strategy,
+            plan_properties,
+        })
+    }
+
+    /// Computes the plan properties based on the table partitioning strategy
+    /// Selects the partitioning strategy based on the table partitioning strategy
+    fn compute_properties(
+        input: &Arc<dyn ExecutionPlan>,
+        partitioning_strategy: &Partitioning,
+    ) -> DFResult<PlanProperties> {
+        let schema = input.schema();
+        let equivalence_properties = EquivalenceProperties::new(schema);
+
+        Ok(PlanProperties::new(
+            equivalence_properties,
+            partitioning_strategy.clone(),
+            EmissionType::Incremental,
+            Boundedness::Bounded,
+        ))
+    }
+
+    /// Determines the optimal partitioning strategy based on table metadata and distribution mode.
+    ///
+    /// This function analyzes the table's partition specification, sort order, and write distribution
+    /// mode to select the most appropriate DataFusion partitioning strategy for insert operations.
+    ///
+    /// ## Distribution Mode Logic
+    ///
+    /// The strategy is determined by the table's `write.distribution-mode` property:
+    ///
+    /// - **`hash` (default)**: Uses hash partitioning on:
+    ///   1. Identity partition columns (e.g., `PARTITIONED BY (year, month)`)
+    ///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+    ///   3. Bucket columns from sort order
+    ///   
+    ///   This ensures data co-location within partitions and buckets for optimal file clustering.
+    ///
+    /// - **`range`**: Approximates range distribution by hashing on sort order columns.
+    ///   Since DataFusion lacks native range exchange, this provides the closest alternative
+    ///   while maintaining some ordering characteristics.
+    ///
+    /// - **`none` or other**: Falls back to round-robin distribution for balanced load.
+    ///
+    /// ## Column Priority and Deduplication
+    ///
+    /// When multiple column sources are available, they are combined in this order:
+    /// 1. Partition identity columns (highest priority)
+    /// 2. Bucket columns from partition spec  
+    /// 3. Bucket columns from sort order
+    /// 4. Sort order columns (for range mode)
+    ///
+    /// Duplicate columns are automatically removed while preserving the priority order.
+    ///
+    /// ## Fallback Behavior
+    ///
+    /// If no suitable hash columns are found (e.g., unpartitioned, non-bucketed table),
+    /// falls back to round-robin batch partitioning for even load distribution.
+    fn determine_partitioning_strategy(

Review Comment:
   > However, this reminds me another case: range only partition, e.g. we only has partitions like date, time. I think in this case we should also use round robin partition since in this case most data are focused in several partitions.
   
   Hum. You are right. The range partitions concentrate data in recent partitions, making hash partitioning counterproductive (considering a date with a temporal partition).
   Since DataFusion doesn't provide Range, the fallback is round-robin and not hashing.
   
   Briefly:
   - Hash partition: Only on bucket columns (partition spec + sort order)
   - Round-robin: Everything else (unpartitioned, range, identity, temporal transforms)
   
   > Also I don't think we should take into account write.distribution-mode for now. The example you use are for spark, but not applicable for datafusion.
   
   Oh, good point, I misunderstood this. I thought it was an iceberg-rust table property.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2318472120


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();
+    let hash_exprs: Vec<Arc<dyn datafusion::physical_expr::PhysicalExpr>> = names_iter
+        .filter(|name| seen.insert(*name))
+        .map(|name| {
+            let idx = input_schema
+                .index_of(name)
+                .map_err(|e| datafusion::error::DataFusionError::Plan(e.to_string()))?;
+            Ok(Arc::new(Column::new(name, idx))

Review Comment:
   Yes!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2318531110


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   > The partition number is irrevalent for writer node. 
   
   Yeah, the actual partition number doesn’t matter to the writer.
   But the mapping from (columns) → (hash value) must be stable across plans, no? That's why I kept the order when creating `Partitioning::Hash`. With different plans, we always have the same order in the hashing columns for the plan.
   What matters is co-location; rows with the same Iceberg partition values must end up together so they can be written to the same data file.
   
   > write data to data files, the writer node needs to calculate and store the partition value of each row, and then send them to partitioned writer.
   
   You’re right, the writer node calculates the partition values for each row and writes them out.
   Before the writer node, though, the partitioning strategy decides how rows with keys are grouped so they end up in the same writer task.
   Are you expecting that partitioning logic to live inside the PartitionedWriter itself?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2313751052


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(

Review Comment:
   True, it will be part of the `ExecutionPlan` building.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2315586822


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();
+    let hash_exprs: Vec<Arc<dyn datafusion::physical_expr::PhysicalExpr>> = names_iter
+        .filter(|name| seen.insert(*name))
+        .map(|name| {
+            let idx = input_schema
+                .index_of(name)
+                .map_err(|e| datafusion::error::DataFusionError::Plan(e.to_string()))?;
+            Ok(Arc::new(Column::new(name, idx))

Review Comment:
   What's I'm saying is that when inserting into a partitioned table in iceberg, the physical plan should looks like
   ```
   WriteExec
         |
   RepartitionExec
        |
   ProjectExec
       |
   TableScanExec      
   ```
   
   Where `ProjectExec` will add an extra column called `_partition`, which is the partition value of each row. And when the `RepartitionExec` chooses to use `HashPartition`, it should repartition using the `_partition` field, rather than the source field.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2314208788


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   I used a `HashSet` because I wanted to keep order of columns.
   If I use an iterator to collect into a `HashSet`, we could lose the order, resulting in incorrect partitioning (iterating in an arbitrary order).
   We could add `indexmap::IndexSet` as a dependency or use 'seen' to maintain the order with a hash set.
   
   In partitioning, order matters: partition identity columns must come before bucket columns. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2306836325


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {

Review Comment:
   All right, let's follow the KISS principle.
   That makes perfect sense. I'll rework that!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2315641228


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   No worries!
   You are right within each plan.
   Let's take an example to see if there's a problem with data consistency if we are iterating without keeping order across two plans.
   
   Data:
   ```
   Row 1: a=5, b=10
   Row 2: a=5, b=10  
   Row 3: a=7, b=3
   ```
   1. HashPartition(vec![a, b], 4)
   ```
   Row 1: hash(5, 10) = 1234 → 1234 % 4 = 2 (goes to partition 2)
   Row 2: hash(5, 10) = 1234 → 1234 % 4 = 2 (goes to partition 2)
   Row 3: hash(7, 3)  = 5678 → 5678 % 4 = 2 (goes to partition 2)
   ```
   2. HashPartition(vec![b, a], 4)
   ```
   Row 1: hash(10, 5) = 8765 → 8765 % 4 = 1 (goes to partition 1)
   Row 2: hash(10, 5) = 8765 → 8765 % 4 = 1 (goes to partition 1)
   Row 3: hash(3, 7)  = 4321 → 4321 % 4 = 1 (goes to partition 1)
   ```
   
   Same data, different distributions:
   Plan 1: Rows 1,2,3 all go to partition 2
   Plan 2: Rows 1,2,3 all go to partition 1
   
   If we are iterating in an arbitrary order on hash partitioning, we could have different plans leading to not ending in the right partition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2315652561


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();
+    let hash_exprs: Vec<Arc<dyn datafusion::physical_expr::PhysicalExpr>> = names_iter
+        .filter(|name| seen.insert(*name))
+        .map(|name| {
+            let idx = input_schema
+                .index_of(name)
+                .map_err(|e| datafusion::error::DataFusionError::Plan(e.to_string()))?;
+            Ok(Arc::new(Column::new(name, idx))

Review Comment:
   Clearer, thanks!
   Indeed, this part needs to be reworked when `ProjectExec` is [merged](https://github.com/apache/iceberg-rust/pull/1602).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2318531110


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   > The partition number is irrevalent for writer node. 
   
   Yeah, the actual partition number doesn’t matter to the writer.
   But the mapping from (columns) → (hash value) must be stable across plan, no? That's why, I kept the order when creating `Partitioning::Hash`. With different plans, we always have the same order in the hashing columns for the plan.
   What matters is co-location; rows with the same Iceberg partition values must end up together so they can be written to the same data file.
   
   > write data to data files, the writer node needs to calculate and store the partition value of each row, and then send them to partitioned writer.
   
   You’re right, the writer node calculates the partition values for each row and writes them out.
   Before the writer node, though, the partitioning strategy decides how rows with keys are grouped so they end up in the same writer task.
   Are you expecting that partitioning logic to live inside the PartitionedWriter itself?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2297894049


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {
+    /// Input execution plan
+    input: Arc<dyn ExecutionPlan>,
+    /// Iceberg table schema to determine partitioning strategy
+    table_schema: SchemaRef,
+    /// Iceberg table metadata to determine partitioning strategy
+    table_metadata: TableMetadataRef,
+    /// Target number of partitions for data distribution
+    target_partitions: usize,
+    /// Partitioning strategy
+    partitioning_strategy: Partitioning,
+    /// Plan properties for optimization
+    plan_properties: PlanProperties,
+}
+
+impl IcebergRepartitionExec {
+    /// Creates a new IcebergRepartitionExec with automatic partitioning strategy selection.
+    ///
+    /// This constructor analyzes the table's partition specification, sort order, and write
+    /// distribution mode to determine the optimal repartitioning strategy for insert operations.
+    ///
+    /// # Arguments
+    ///
+    /// * `input` - The input execution plan providing data to be repartitioned
+    /// * `table_schema` - The Iceberg table schema used to resolve column references
+    /// * `table_metadata` - The Iceberg table metadata containing partition spec, sort order,
+    ///   and table properties including write distribution mode
+    /// * `target_partitions` - Target number of partitions for parallel processing:
+    ///   - `0`: Auto-detect from DataFusion's SessionConfig target_partitions (recommended)
+    ///   - `> 0`: Use explicit partition count for specific performance tuning
+    ///
+    /// # Returns
+    ///
+    /// A configured repartition execution plan that will apply the optimal partitioning
+    /// strategy during execution, or pass through unchanged data if no repartitioning
+    /// is needed.
+    ///
+    /// # Example
+    ///
+    /// ```ignore
+    /// let repartition_exec = IcebergRepartitionExec::new(
+    ///     input_plan,
+    ///     table.schema_ref(),
+    ///     table.metadata_ref(),
+    ///     state.config().target_partitions(),
+    /// )?;
+    /// ```
+    pub fn new(
+        input: Arc<dyn ExecutionPlan>,
+        table_schema: SchemaRef,
+        table_metadata: TableMetadataRef,
+        target_partitions: usize,
+    ) -> DFResult<Self> {
+        if target_partitions == 0 {
+            return Err(datafusion::error::DataFusionError::Plan(
+                "IcebergRepartitionExec requires target_partitions > 0".to_string(),
+            ));
+        }
+
+        let partitioning_strategy = Self::determine_partitioning_strategy(
+            &input,
+            &table_schema,
+            &table_metadata,
+            target_partitions,
+        )?;
+
+        let plan_properties = Self::compute_properties(&input, &partitioning_strategy)?;
+
+        Ok(Self {
+            input,
+            table_schema,
+            table_metadata,
+            target_partitions,
+            partitioning_strategy,
+            plan_properties,
+        })
+    }
+
+    /// Computes the plan properties based on the table partitioning strategy
+    /// Selects the partitioning strategy based on the table partitioning strategy
+    fn compute_properties(
+        input: &Arc<dyn ExecutionPlan>,
+        partitioning_strategy: &Partitioning,
+    ) -> DFResult<PlanProperties> {
+        let schema = input.schema();
+        let equivalence_properties = EquivalenceProperties::new(schema);
+
+        Ok(PlanProperties::new(
+            equivalence_properties,
+            partitioning_strategy.clone(),
+            EmissionType::Incremental,
+            Boundedness::Bounded,
+        ))
+    }
+
+    /// Determines the optimal partitioning strategy based on table metadata and distribution mode.
+    ///
+    /// This function analyzes the table's partition specification, sort order, and write distribution
+    /// mode to select the most appropriate DataFusion partitioning strategy for insert operations.
+    ///
+    /// ## Distribution Mode Logic
+    ///
+    /// The strategy is determined by the table's `write.distribution-mode` property:
+    ///
+    /// - **`hash` (default)**: Uses hash partitioning on:
+    ///   1. Identity partition columns (e.g., `PARTITIONED BY (year, month)`)
+    ///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+    ///   3. Bucket columns from sort order
+    ///   
+    ///   This ensures data co-location within partitions and buckets for optimal file clustering.
+    ///
+    /// - **`range`**: Approximates range distribution by hashing on sort order columns.
+    ///   Since DataFusion lacks native range exchange, this provides the closest alternative
+    ///   while maintaining some ordering characteristics.
+    ///
+    /// - **`none` or other**: Falls back to round-robin distribution for balanced load.
+    ///
+    /// ## Column Priority and Deduplication
+    ///
+    /// When multiple column sources are available, they are combined in this order:
+    /// 1. Partition identity columns (highest priority)
+    /// 2. Bucket columns from partition spec  
+    /// 3. Bucket columns from sort order
+    /// 4. Sort order columns (for range mode)
+    ///
+    /// Duplicate columns are automatically removed while preserving the priority order.
+    ///
+    /// ## Fallback Behavior
+    ///
+    /// If no suitable hash columns are found (e.g., unpartitioned, non-bucketed table),
+    /// falls back to round-robin batch partitioning for even load distribution.
+    fn determine_partitioning_strategy(

Review Comment:
   > However, this reminds me another case: range only partition, e.g. we only has partitions like date, time. I think in this case we should also use round robin partition since in this case most data are focused in several partitions.
   
   Hum. You are right. The range partitions concentrate data in recent partitions, making hash partitioning counterproductive (considering a date with a temporal partition).
   Since DataFusion doesn't provide Range, the fallback is round-robin and not hashing.
   
   Briefly:
   - Hash partition: Only on bucket columns (partition spec + sort order)
   - Round-robin: Everything else (unpartitioned, range, identity, temporal transforms)
   
   > Also I don't think we should take into account write.distribution-mode for now. The example you use are for spark, but not applicable for datafusion.
   Oh, good point, I misunderstood this. I thought it was an icebergrust table property.



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {
+    /// Input execution plan
+    input: Arc<dyn ExecutionPlan>,
+    /// Iceberg table schema to determine partitioning strategy
+    table_schema: SchemaRef,
+    /// Iceberg table metadata to determine partitioning strategy
+    table_metadata: TableMetadataRef,
+    /// Target number of partitions for data distribution
+    target_partitions: usize,
+    /// Partitioning strategy
+    partitioning_strategy: Partitioning,
+    /// Plan properties for optimization
+    plan_properties: PlanProperties,
+}
+
+impl IcebergRepartitionExec {
+    /// Creates a new IcebergRepartitionExec with automatic partitioning strategy selection.
+    ///
+    /// This constructor analyzes the table's partition specification, sort order, and write
+    /// distribution mode to determine the optimal repartitioning strategy for insert operations.
+    ///
+    /// # Arguments
+    ///
+    /// * `input` - The input execution plan providing data to be repartitioned
+    /// * `table_schema` - The Iceberg table schema used to resolve column references
+    /// * `table_metadata` - The Iceberg table metadata containing partition spec, sort order,
+    ///   and table properties including write distribution mode
+    /// * `target_partitions` - Target number of partitions for parallel processing:
+    ///   - `0`: Auto-detect from DataFusion's SessionConfig target_partitions (recommended)
+    ///   - `> 0`: Use explicit partition count for specific performance tuning
+    ///
+    /// # Returns
+    ///
+    /// A configured repartition execution plan that will apply the optimal partitioning
+    /// strategy during execution, or pass through unchanged data if no repartitioning
+    /// is needed.
+    ///
+    /// # Example
+    ///
+    /// ```ignore
+    /// let repartition_exec = IcebergRepartitionExec::new(
+    ///     input_plan,
+    ///     table.schema_ref(),
+    ///     table.metadata_ref(),
+    ///     state.config().target_partitions(),
+    /// )?;
+    /// ```
+    pub fn new(
+        input: Arc<dyn ExecutionPlan>,
+        table_schema: SchemaRef,
+        table_metadata: TableMetadataRef,
+        target_partitions: usize,
+    ) -> DFResult<Self> {
+        if target_partitions == 0 {
+            return Err(datafusion::error::DataFusionError::Plan(
+                "IcebergRepartitionExec requires target_partitions > 0".to_string(),
+            ));
+        }
+
+        let partitioning_strategy = Self::determine_partitioning_strategy(
+            &input,
+            &table_schema,
+            &table_metadata,
+            target_partitions,
+        )?;
+
+        let plan_properties = Self::compute_properties(&input, &partitioning_strategy)?;
+
+        Ok(Self {
+            input,
+            table_schema,
+            table_metadata,
+            target_partitions,
+            partitioning_strategy,
+            plan_properties,
+        })
+    }
+
+    /// Computes the plan properties based on the table partitioning strategy
+    /// Selects the partitioning strategy based on the table partitioning strategy
+    fn compute_properties(
+        input: &Arc<dyn ExecutionPlan>,
+        partitioning_strategy: &Partitioning,
+    ) -> DFResult<PlanProperties> {
+        let schema = input.schema();
+        let equivalence_properties = EquivalenceProperties::new(schema);
+
+        Ok(PlanProperties::new(
+            equivalence_properties,
+            partitioning_strategy.clone(),
+            EmissionType::Incremental,
+            Boundedness::Bounded,
+        ))
+    }
+
+    /// Determines the optimal partitioning strategy based on table metadata and distribution mode.
+    ///
+    /// This function analyzes the table's partition specification, sort order, and write distribution
+    /// mode to select the most appropriate DataFusion partitioning strategy for insert operations.
+    ///
+    /// ## Distribution Mode Logic
+    ///
+    /// The strategy is determined by the table's `write.distribution-mode` property:
+    ///
+    /// - **`hash` (default)**: Uses hash partitioning on:
+    ///   1. Identity partition columns (e.g., `PARTITIONED BY (year, month)`)
+    ///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+    ///   3. Bucket columns from sort order
+    ///   
+    ///   This ensures data co-location within partitions and buckets for optimal file clustering.
+    ///
+    /// - **`range`**: Approximates range distribution by hashing on sort order columns.
+    ///   Since DataFusion lacks native range exchange, this provides the closest alternative
+    ///   while maintaining some ordering characteristics.
+    ///
+    /// - **`none` or other**: Falls back to round-robin distribution for balanced load.
+    ///
+    /// ## Column Priority and Deduplication
+    ///
+    /// When multiple column sources are available, they are combined in this order:
+    /// 1. Partition identity columns (highest priority)
+    /// 2. Bucket columns from partition spec  
+    /// 3. Bucket columns from sort order
+    /// 4. Sort order columns (for range mode)
+    ///
+    /// Duplicate columns are automatically removed while preserving the priority order.
+    ///
+    /// ## Fallback Behavior
+    ///
+    /// If no suitable hash columns are found (e.g., unpartitioned, non-bucketed table),
+    /// falls back to round-robin batch partitioning for even load distribution.
+    fn determine_partitioning_strategy(

Review Comment:
   > However, this reminds me another case: range only partition, e.g. we only has partitions like date, time. I think in this case we should also use round robin partition since in this case most data are focused in several partitions.
   
   Hum. You are right. The range partitions concentrate data in recent partitions, making hash partitioning counterproductive (considering a date with a temporal partition).
   Since DataFusion doesn't provide Range, the fallback is round-robin and not hashing.
   
   Briefly:
   - Hash partition: Only on bucket columns (partition spec + sort order)
   - Round-robin: Everything else (unpartitioned, range, identity, temporal transforms)
   
   > Also I don't think we should take into account write.distribution-mode for now. The example you use are for spark, but not applicable for datafusion.
   
   Oh, good point, I misunderstood this. I thought it was an icebergrust table property.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2314208788


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   I used a `HashSet` because I wanted to keep order.
   If I use an iterator to collect into a `HashSet`, we could lose the order, resulting in incorrect partitioning.
   We could add `indexmap::IndexSet` as a dependency or use 'seen' to maintain the order with a hash set.
   
   In partitioning, order matters: partition identity columns must come before bucket columns. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2469684241


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,966 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering. This is used when the `_partition`
+///   column is present AND the partition spec has hash-friendly transforms (Identity/Bucket),
+///   or when source columns with these transforms are available.
+/// - **Round-robin partitioning** – Used for temporal transforms (Year, Month, Day, Hour),
+///   Truncate, or other transforms that don't provide good hash distribution.
+/// - **Mixed transforms** – Combines multiple transform types, using hash partitioning only
+///   when Identity or Bucket transforms are present, otherwise falling back to round-robin.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned. For partitioned tables,
+///   the input should include the `_partition` column (added via `project_with_partition`).
+/// * `table_metadata` – Iceberg table metadata containing partition spec.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+#[allow(dead_code)]
+pub(crate) fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {

Review Comment:
   The benefit of this check is to avoid redundant repartitioning when the input already has correct partitioning, when it's called multiple times, or during optimization passes.
   Since DataFusion normally builds the plan once (no manual calls), it's okay to remove it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2324709443


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   > Please not that the partition_value_of is calculated using iceberg's transform function, rather than datafusion's partition function.
   
   Ah, this might be the source of confusion!
   
   For the Hashing function:
   - Using DataFusion and a dedicated node to handle the partitioning strategy with `RepartitionExec`, the column order matters for DataFusion implementation using its Hashing partition function.
   Datafusion will hash the partitions' values to distribute for the node:
   `RepartitionExec(HashPartition(vec![a, b]))` and `RepartitionExec(HashPartition(vec![b, a]))` are not equivalent from DataFusion's perspective.
   
   - If we use our own `partition_value_of(row)`, the orders could not matter if we use a hashing function that is not order-dependent.
   
   Coming back to your original comment, I used a dedicated node for the execution plan, that's why I focused on keeping the column order:
   - **DataFusion's Execution Partitioning:** For the query optimizer to recognize that two data streams are partitioned compatibly (e.g., for a join), they must have the same canonical Partitioning scheme. Using an unstable order from a HashSet would break this, leading to inefficient plans.
   - **Iceberg's Storage Partitioning**: This happens after the DataFusion shuffle. Once the data is co-located on a worker node, the writer then uses Iceberg's transform function (partition_value_of) to determine the final storage path.
   
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2318442343


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   > If we are iterating in an arbitrary order on hash partitioning, we could have different plans leading to not ending in the right partition.
   
   I still don't get the point. The partition number is irrevalent for writer node. For partitioned table, when write data to data files, the writer node needs to calculate and store the partition value of each row, and then send them to partitioned writer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2297880427


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {

Review Comment:
   Thank you for your comments!
   
   `RepartitionExec` is a generic operator: it reshuffles rows based on a distribution requirement.
   Iceberg has stricter requirements for how data must be partitioned, bucketed, sorted, and grouped before writing. So, we must select the relevant partition strategy.
   
   We have special requirements before writing:
   1. Use Iceberg table metadata:
   - Partition specifications (identity transforms, bucket transforms)
   - Sort orders
   - Write distribution mode (hash, range, none)
   2. Select the appropriate partitioning strategy:
   - Hash partitioning on partition/bucket columns for partitioned tables
   - Round-robin for unpartitioned tables
   - Range approximation using sort order columns
   
   Some other requirements to preserve the" partition–bucket–range ordering" semantics required by Iceberg:
   - Partition columns must be respected in the physical file layout
   - Bucketing/range partitioning needs to be reproducible and consistent
   - File grouping must align with Iceberg metadata expectations
   
   Repartitioning is a plan-level operator, not an expression:
   - `PhysicalExpr` can help compute the partition/bucket key for a row.
   - Reshuffling rows into partitions is still an execution node (ExecutionPlan).
   - If we only extend `PhysicalExpr`, we'll have an expression that can calculate partition/bucket values, but we still need an Exec node to do the actual shuffle/repartitioning.
   
   So, in a nutshell, why we need our "Iceberg-aware" strategy  (`IcebergRepartitionExec`) to determine the best partitioning, and we use it for Datafusion (calling `RepartitionExec` with our selection), and we use `PhysicalExpr` for determining it: 
   
   ```
   IcebergRepartitionExec (strategy selection, Iceberg-aware)
       ↳ chooses partitioning (hash/round-robin/range)
       ↳ uses Iceberg metadata (partition spec, sort order, mode)
       ↓
   DataFusion RepartitionExec (generic shuffle operator)
       ↳ actually reshuffles rows into partitions
       ↓
   PhysicalExpr (partition/bucket key computation)
       ↳ hash/range/bucket expressions evaluated per row
   ```
   
   Of course, if we decide to rely 100% on DataFusion, we need to consider:
   - `RepartitionExec` implements generic distributions without understanding the Iceberg specificities (bucket, partitions, range vs sort)
   - Iceberg requires that bucketing and range partitioning be reproducible and consistent across writers
   - Iceberg expects hierarchical ordering: partition → bucket → range
   - Data  Inconsistency risk? may not be reproductible?
   - If Iceberg semantics aren’t enforced at write time, we will need extra cleanup/repair jobs later (e.g., repartitioning files offline or rewriting manifests for metadata) or custom implementation
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2303537024


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,906 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::any::Any;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::execution::{SendableRecordBatchStream, TaskContext};
+use datafusion::physical_expr::{EquivalenceProperties, PhysicalExpr};
+use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType};
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{
+    DisplayAs, DisplayFormatType, ExecutionPlan, Partitioning, PlanProperties,
+};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Iceberg-specific repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This execution plan automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and the configured write distribution mode:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Partitioned tables**: Uses hash partitioning on partition columns (identity transforms)
+///   and bucket columns to maintain data co-location. This ensures:
+///   - Better file clustering within partitions
+///   - Improved query pruning performance
+///   - Optimal join performance on partitioned columns
+///
+/// - **Range-distributed tables**: Approximates range distribution by hashing on sort order
+///   columns since DataFusion lacks native range exchange. Falls back to partition/bucket
+///   column hashing when available.
+///
+/// ## Write Distribution Modes
+///
+/// Respects the table's `write.distribution-mode` property:
+/// - `hash` (default): Distributes by partition and bucket columns
+/// - `range`: Distributes by sort order columns
+/// - `none`: Uses round-robin distribution
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Automatically detects optimal partition count from DataFusion's SessionConfig
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+#[derive(Debug)]
+pub struct IcebergRepartitionExec {

Review Comment:
   I agree the point that we need to determine the partition strategy based on iceberg's table partitioning, but that doesn't mean we need to implement a new executor.  What I was saying is  why not just keep `determine_partitioning_strategy` to determine the partition, and have a utility function to create a `RepartitionExec`? 
   Also your implementation is just a wrapper of `RepartitionExec` IIUC.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2480528735


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,966 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering. This is used when the `_partition`
+///   column is present AND the partition spec has hash-friendly transforms (Identity/Bucket),
+///   or when source columns with these transforms are available.
+/// - **Round-robin partitioning** – Used for temporal transforms (Year, Month, Day, Hour),
+///   Truncate, or other transforms that don't provide good hash distribution.
+/// - **Mixed transforms** – Combines multiple transform types, using hash partitioning only
+///   when Identity or Bucket transforms are present, otherwise falling back to round-robin.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned. For partitioned tables,
+///   the input should include the `_partition` column (added via `project_with_partition`).
+/// * `table_metadata` – Iceberg table metadata containing partition spec.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+#[allow(dead_code)]
+pub(crate) fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {

Review Comment:
   I agree that this seems a redundant optimization which could be done in datafusion, I would prefer to remove it for now to keep things simple first.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2314237798


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {

Review Comment:
   Agreed.
   Indeed, sorting is inside a partition and doesn't require a partitioning strategy.
   I will fix this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2314437310


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();
+    let hash_exprs: Vec<Arc<dyn datafusion::physical_expr::PhysicalExpr>> = names_iter
+        .filter(|name| seen.insert(*name))
+        .map(|name| {
+            let idx = input_schema
+                .index_of(name)
+                .map_err(|e| datafusion::error::DataFusionError::Plan(e.to_string()))?;
+            Ok(Arc::new(Column::new(name, idx))

Review Comment:
   Adding more information to the error log.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2315574258


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   > In partitioning, order matters: partition identity columns must come before bucket columns.
   
   Sorry, I still not don't understand why order matters.  I think the following two plans
   ```
   RepartitionExec(HashPartition(vec![a, b], 4))
                            | 
   TableScan([a, b, c, d])
   ```
   
   ```
   RepartitionExec(HashPartition(vec![b, a], 4))
                            | 
   TableScan([a, b, c, d])
   ```
   
   are same? 
   
   Both of them guarantee that if two rows have same value of column a and b, they should go into same partition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2315641228


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_schema: SchemaRef,
+    table_metadata: TableMetadataRef,
+    target_partitions: usize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    if target_partitions == 0 {
+        return Err(datafusion::error::DataFusionError::Plan(
+            "repartition requires target_partitions > 0".to_string(),
+        ));
+    }
+
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_schema, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(
+    a_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+    b_exprs: &[Arc<dyn datafusion::physical_expr::PhysicalExpr>],
+) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs
+        .iter()
+        .zip(b_exprs.iter())
+        .all(|(a, b)| a.as_any().downcast_ref::<Column>() == b.as_any().downcast_ref::<Column>())
+}
+
+/// Determines the optimal partitioning strategy based on table metadata.
+///
+/// This function analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy Logic
+///
+/// The strategy is determined by analyzing the table's partition transforms:
+///
+/// - **Hash partitioning**: Used only when there are identity transforms (direct column partitioning)
+///   or bucket transforms that provide good data distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order
+///   
+///   This ensures data co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Unpartitioned tables
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Tables with no suitable hash columns
+///
+/// ## Column Priority and Deduplication
+///
+/// When multiple column sources are available, they are combined in this order:
+/// 1. Partition identity columns (highest priority)
+/// 2. Bucket columns from partition spec  
+/// 3. Bucket columns from sort order
+///
+/// Duplicate columns are automatically removed while preserving the priority order.
+///
+/// ## Fallback Behavior
+///
+/// If no suitable hash columns are found (e.g., unpartitioned, range-only, or non-bucketed table),
+/// falls back to round-robin batch partitioning for even load distribution.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_schema: &SchemaRef,
+    table_metadata: &TableMetadata,
+    target_partitions: usize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let sort_order = table_metadata.default_sort_order();
+
+    let names_iter: Box<dyn Iterator<Item = &str>> = {
+        // Partition identity columns
+        let part_names = partition_spec.fields().iter().filter_map(|pf| {
+            if matches!(pf.transform, Transform::Identity) {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from partition spec
+        let bucket_names_part = partition_spec.fields().iter().filter_map(|pf| {
+            if let Transform::Bucket(_) = pf.transform {
+                table_schema
+                    .field_by_id(pf.source_id)
+                    .map(|sf| sf.name.as_str())
+            } else {
+                None
+            }
+        });
+        // Bucket columns from sort order
+        let bucket_names_sort = sort_order.fields.iter().filter_map(|sf| {
+            if let Transform::Bucket(_) = sf.transform {
+                table_schema
+                    .field_by_id(sf.source_id)
+                    .map(|field| field.name.as_str())
+            } else {
+                None
+            }
+        });
+        Box::new(part_names.chain(bucket_names_part).chain(bucket_names_sort))
+    };
+
+    // Order: partitions first, then buckets
+    // Deduplicate while preserving order
+    let input_schema = input.schema();
+    let mut seen: HashSet<&str> = HashSet::new();

Review Comment:
   No worries!
   You are right.
   Let's take an example to see if there's a problem with data consistency if we are iterating without keeping order across two plans.
   
   Data:
   ```
   Row 1: a=5, b=10
   Row 2: a=5, b=10  
   Row 3: a=7, b=3
   ```
   1. HashPartition(vec![a, b], 4)
   ```
   Row 1: hash(5, 10) = 1234 → 1234 % 4 = 2 (goes to partition 2)
   Row 2: hash(5, 10) = 1234 → 1234 % 4 = 2 (goes to partition 2)
   Row 3: hash(7, 3)  = 5678 → 5678 % 4 = 2 (goes to partition 2)
   ```
   2. HashPartition(vec![b, a], 4)
   ```
   Row 1: hash(10, 5) = 8765 → 8765 % 4 = 1 (goes to partition 1)
   Row 2: hash(10, 5) = 8765 → 8765 % 4 = 1 (goes to partition 1)
   Row 3: hash(3, 7)  = 4321 → 4321 % 4 = 1 (goes to partition 1)
   ```
   
   Same data, different distributions:
   Plan 1: Rows 1,2,3 all go to partition 2
   Plan 2: Rows 1,2,3 all go to partition 1
   
   If we are iterating in an arbitrary order on hash partitioning, we could have different plans leading to not ending in the right partition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

Posted by "fvaleye (via GitHub)" <gi...@apache.org>.
fvaleye commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2465152295


##########
crates/iceberg/src/table.rs:
##########
@@ -235,6 +235,11 @@ impl Table {
         self.readonly
     }
 
+    /// Returns the current schema as a shared reference.
+    pub fn schema_ref(&self) -> SchemaRef {

Review Comment:
   Fixed!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org