You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "liurenjie1024 (via GitHub)" <gi...@apache.org> on 2025/10/27 09:59:43 UTC

Re: [PR] feat(datafusion): implement the partitioning node for DataFusion to define the partitioning [iceberg-rust]

liurenjie1024 commented on code in PR #1620:
URL: https://github.com/apache/iceberg-rust/pull/1620#discussion_r2465061090


##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,798 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::num::NonZeroUsize;
+use std::sync::Arc;
+
+use datafusion::error::{DataFusionError, Result as DFResult};
+use datafusion::physical_expr::PhysicalExpr;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{TableMetadata, TableMetadataRef, Transform};
+use tracing;
+
+use crate::physical_plan::project::PARTITION_VALUES_COLUMN;
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// Automatically determines the optimal partitioning strategy based on the table's
+/// partition specification and sort order.
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables** – Uses round-robin distribution to balance load across workers.
+/// - **Hash partitioning** – Applied for tables with identity or bucket transforms, ensuring
+///   co-location of related data for efficient file clustering.
+/// - **Round-robin partitioning** – Used for range-only or temporal transforms that don’t
+///   provide good distribution.
+/// - **Mixed transforms** – Combines range and identity/bucket transforms, using hash
+///   partitioning on identity/bucket columns.
+///
+/// ## Performance Notes
+///
+/// - Only repartitions when the input scheme or partition count differs from the target.
+/// - Requires an explicit target partition count for deterministic behavior.
+/// - Preserves column order (partitions first, then buckets) for consistent layout.
+///
+/// # Arguments
+///
+/// * `input` – The input execution plan providing data to be repartitioned (already projected to match table schema).
+/// * `table_metadata` – Iceberg table metadata containing partition spec and sort order.
+/// * `target_partitions` – Target number of partitions for parallel processing (must be > 0).
+///
+/// # Returns
+///
+/// An execution plan that applies the optimal partitioning strategy, or the original input plan if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(
+    input: Arc<dyn ExecutionPlan>,
+    table_metadata: TableMetadataRef,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Arc<dyn ExecutionPlan>> {
+    let partitioning_strategy =
+        determine_partitioning_strategy(&input, &table_metadata, target_partitions)?;
+
+    if !needs_repartitioning(&input, &partitioning_strategy) {
+        return Ok(input);
+    }
+
+    Ok(Arc::new(RepartitionExec::try_new(
+        input,
+        partitioning_strategy,
+    )?))
+}
+
+/// Returns whether repartitioning is actually needed by comparing input and desired partitioning
+fn needs_repartitioning(input: &Arc<dyn ExecutionPlan>, desired: &Partitioning) -> bool {
+    let input_partitioning = input.properties().output_partitioning();
+    match (input_partitioning, desired) {
+        (Partitioning::RoundRobinBatch(a), Partitioning::RoundRobinBatch(b)) => a != b,
+        (Partitioning::Hash(a_exprs, a_n), Partitioning::Hash(b_exprs, b_n)) => {
+            a_n != b_n || !same_columns(a_exprs, b_exprs)
+        }
+        _ => true,
+    }
+}
+
+/// Helper function to check if two sets of column expressions are the same
+fn same_columns(a_exprs: &[Arc<dyn PhysicalExpr>], b_exprs: &[Arc<dyn PhysicalExpr>]) -> bool {
+    if a_exprs.len() != b_exprs.len() {
+        return false;
+    }
+    a_exprs.iter().zip(b_exprs.iter()).all(|(a, b)| {
+        if let (Some(a_col), Some(b_col)) = (
+            a.as_any().downcast_ref::<Column>(),
+            b.as_any().downcast_ref::<Column>(),
+        ) {
+            a_col.name() == b_col.name() && a_col.index() == b_col.index()
+        } else {
+            std::ptr::eq(a.as_ref(), b.as_ref())
+        }
+    })
+}
+
+/// Determine the optimal partitioning strategy based on table metadata.
+///
+/// Analyzes the table's partition specification and sort order to select
+/// the most appropriate DataFusion partitioning strategy for insert operations.
+///
+/// ## Partitioning Strategy
+///
+/// - **Hash partitioning using `_partition` column**: Used when the input includes a
+///   projected `_partition` column. Ensures data is distributed based on actual partition values.
+///
+/// - **Hash partitioning using source columns**: Applied when identity or bucket transforms
+///   provide good distribution:
+///   1. Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   2. Bucket columns from partition spec (e.g., `bucket(16, user_id)`)
+///   3. Bucket columns from sort order  
+///   Ensures co-location within partitions and buckets for optimal file clustering.
+///
+/// - **Round-robin partitioning**: Used for unpartitioned, range-only, or non-bucketed tables
+///   where hash partitioning is not feasible.
+///
+/// ## Column Priority
+///
+/// Columns are combined in the following order, with duplicates removed:
+/// 1. `_partition` column (highest priority, if present)
+/// 2. Identity partition columns
+/// 3. Bucket columns from partition spec
+/// 4. Bucket columns from sort order
+///
+/// ## Fallback
+///
+/// If no suitable hash columns are found, falls back to round-robin batch partitioning
+/// to ensure even load distribution across partitions.
+fn determine_partitioning_strategy(
+    input: &Arc<dyn ExecutionPlan>,
+    table_metadata: &TableMetadata,
+    target_partitions: NonZeroUsize,
+) -> DFResult<Partitioning> {
+    let partition_spec = table_metadata.default_partition_spec();
+    let table_schema = table_metadata.current_schema();
+    let input_schema = input.schema();
+
+    match (
+        !partition_spec.is_unpartitioned(),
+        input_schema.index_of(PARTITION_VALUES_COLUMN),
+    ) {
+        (true, Ok(partition_col_idx)) => {
+            let partition_field = input_schema.field(partition_col_idx);
+            if partition_field.name() != PARTITION_VALUES_COLUMN {
+                return Err(DataFusionError::Plan(format!(
+                    "Expected {} column at index {}, but found '{}'",
+                    PARTITION_VALUES_COLUMN,
+                    partition_col_idx,
+                    partition_field.name()
+                )));
+            }
+
+            let partition_expr = Arc::new(Column::new(PARTITION_VALUES_COLUMN, partition_col_idx))
+                as Arc<dyn PhysicalExpr>;
+            return Ok(Partitioning::Hash(

Review Comment:
   This is incorrect? We should move the logic to determine hash or range here?



##########
crates/integrations/datafusion/src/physical_plan/repartition.rs:
##########
@@ -0,0 +1,805 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::HashSet;
+use std::sync::Arc;
+
+use datafusion::error::Result as DFResult;
+use datafusion::physical_plan::expressions::Column;
+use datafusion::physical_plan::repartition::RepartitionExec;
+use datafusion::physical_plan::{ExecutionPlan, Partitioning};
+use iceberg::spec::{SchemaRef, TableMetadata, TableMetadataRef, Transform};
+
+/// Creates an Iceberg-aware repartition execution plan that optimizes data distribution
+/// for parallel processing while respecting Iceberg table partitioning semantics.
+///
+/// This function automatically determines the optimal partitioning strategy based on
+/// the table's partition specification and sort order:
+///
+/// ## Partitioning Strategies
+///
+/// - **Unpartitioned tables**: Uses round-robin distribution to ensure balanced load
+///   across all workers, maximizing parallelism for write operations.
+///
+/// - **Hash partitioning**: Used for tables with identity transforms or bucket transforms:
+///   - Identity partition columns (e.g., `PARTITIONED BY (user_id, category)`)
+///   - Bucket columns from partition spec or sort order
+///   - This ensures data co-location within partitions and buckets for optimal file clustering
+///
+/// - **Round-robin partitioning**: Used for:
+///   - Range-only partitions (e.g., date/time partitions that concentrate data)
+///   - Tables with only temporal/range transforms that don't provide good distribution
+///   - Unpartitioned or non-bucketed tables
+///
+/// - **Mixed transforms**: Tables with both range and identity/bucket transforms use hash
+///   partitioning on the identity/bucket columns for optimal distribution.
+///
+/// ## Performance notes
+///
+/// - Only repartitions when the input partitioning scheme differs from the desired strategy
+/// - Only repartitions when the input partition count differs from the target
+/// - Requires explicit target partition count for deterministic behavior
+/// - Preserves column order (partitions first, then buckets) for consistent file layout
+///
+/// # Arguments
+///
+/// * `input` - The input execution plan providing data to be repartitioned
+/// * `table_schema` - The Iceberg table schema used to resolve column references
+/// * `table_metadata` - The Iceberg table metadata containing partition spec and sort order
+/// * `target_partitions` - Target number of partitions for parallel processing (must be > 0)
+///
+/// # Returns
+///
+/// An execution plan that will apply the optimal partitioning strategy during execution,
+/// or the original input plan unchanged if no repartitioning is needed.
+///
+/// # Example
+///
+/// ```ignore
+/// let repartitioned_plan = repartition(
+///     input_plan,
+///     table.schema_ref(),
+///     table.metadata_ref(),
+///     4, // Explicit partition count
+/// )?;
+/// ```
+pub fn repartition(

Review Comment:
   This is not resolved?



##########
crates/iceberg/src/table.rs:
##########
@@ -235,6 +235,11 @@ impl Table {
         self.readonly
     }
 
+    /// Returns the current schema as a shared reference.
+    pub fn schema_ref(&self) -> SchemaRef {

Review Comment:
   ```suggestion
       pub fn current_schema_ref(&self) -> SchemaRef {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org