You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "hotcache (via GitHub)" <gi...@apache.org> on 2026/08/23 05:26:54 UTC

[PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

hotcache opened a new pull request, #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046

   ## Which issue does this PR close?
   
     Part of #1607.
   
     This is the first in a series of PRs to add compaction/rewrite support
     to iceberg-rust. Follow-up PRs will add manifest caching, OverwriteFiles,
     RowDelta, and conflict detection.
   
   ## What changes are included in this PR?
   
   Introduces `MergingSnapshotProducer` and `RewriteFilesAction` — the Rust
   equivalents of Java's `MergingSnapshotProducer` and `BaseRewriteFiles`.
   
   **MergingSnapshotProducer** (`merging.rs`) is a shared engine for
   snapshot-producing operations that both add and delete data files:
   - `ManifestFilterManager` rewrites existing manifests to remove deleted
     file entries, supporting partition-evolved tables (non-default specs)
   - Writes new manifests for added files
   - Computes snapshot summaries accounting for both additions and removals
   - Delegates final snapshot/manifest-list creation to `SnapshotProducer`
   
   **RewriteFilesAction** (`rewrite.rs`) is a thin action on top of
   `MergingSnapshotProducer` that replaces data files (compaction). It
   validates that at least one file is deleted and one added, and commits
   with `Operation::Replace`.
   
   ### Design
   
   As discussed in #2620, `MergingSnapshotProducer` uses composition
   rather than inheritance — each action owns an instance and configures it
   with its own operation type and validation rules. This aligns with
   @JanKaul's [recent analysis](https://github.com/apache/iceberg-rust/pull/2620#issuecomment-5366874140)
   that the MergeCache-per-action approach converges to the same struct
   field-for-field.
   
   ### Planned follow-ups
   
   - **PR2**: Manifest cache for retry optimization
   - **PR3**: `OverwriteFilesAction` with explicit file mode
   - **PR4**: `overwrite_by_row_filter(Expression)` for retry-safe overwrites
   - **PR5**: `RowDeltaAction` for position/equality delete files
   - **PR6**: Snapshot validation (conflict detection)
   
   ## Are these changes tested?
   
   7 tests covering:
   - E2E compaction (3 files → 1 merged)
   - Partial rewrite (2 of 3 files)
   - Missing delete target → error
   - Empty table rewrite → error
   - Empty manifest omission
   - Validation: no deletes → error, no adds → error
   
   ## AI Disclosure
   
   AI (Claude) was used to assist with code review, test generation, and
   PR description drafting. All code was reviewed and validated by the author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "jopdorp (via GitHub)" <gi...@apache.org>.
jopdorp commented on PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#issuecomment-6519496604

   @laskoviymishka @JanKaul opened #3585, which carries @hotcache's commits as they were, merged with main, plus the remove-only rewrite from hotcache/iceberg-rust#3. So this one can be closed in favour of it, and we're happy to hand it back if @hotcache picks it up again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "hotcache (via GitHub)" <gi...@apache.org>.
hotcache commented on code in PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#discussion_r3849138698


##########
crates/iceberg/src/transaction/merging.rs:
##########
@@ -0,0 +1,398 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Shared engine for snapshot-producing operations that both add and delete files.
+//!
+//! [`MergingSnapshotProducer`] is the Rust equivalent of Java's
+//! `MergingSnapshotProducer`. It handles manifest filtering, new manifest
+//! creation, summary computation, and delegates the final snapshot commit to
+//! [`SnapshotProducer`].
+
+use std::collections::{HashMap, HashSet};
+
+use uuid::Uuid;
+
+use crate::error::Result;
+use crate::io::OutputFile;
+use crate::spec::{
+    DataContentType, DataFile, DataFileFormat, FormatVersion, ManifestContentType, ManifestEntry,
+    ManifestFile, ManifestStatus, ManifestWriter, ManifestWriterBuilder, Operation, PartitionSpec,
+    SchemaRef, SnapshotSummaryCollector, Summary, update_snapshot_summaries,
+};
+use crate::table::Table;
+use crate::transaction::ActionCommit;
+use crate::transaction::snapshot::SnapshotProducer;
+use crate::{Error, ErrorKind};
+
+/// Create a manifest writer that handles encryption when available.
+fn new_manifest_writer(
+    table: &Table,
+    output_file: OutputFile,
+    snapshot_id: Option<i64>,
+    content: ManifestContentType,
+    schema: SchemaRef,
+    partition_spec: PartitionSpec,
+) -> Result<ManifestWriter> {
+    let builder = if let Some(em) = table.encryption_manager() {
+        ManifestWriterBuilder::new_from_encrypted(
+            em.encrypt(output_file),
+            snapshot_id,
+            schema,
+            partition_spec,
+        )?
+    } else {
+        ManifestWriterBuilder::new(output_file, snapshot_id, schema, partition_spec)
+    };
+
+    match table.metadata().format_version() {
+        FormatVersion::V1 => Ok(builder.build_v1()),
+        FormatVersion::V2 => match content {
+            ManifestContentType::Data => Ok(builder.build_v2_data()),
+            ManifestContentType::Deletes => Ok(builder.build_v2_deletes()),
+        },
+        FormatVersion::V3 => match content {
+            ManifestContentType::Data => Ok(builder.build_v3_data()),
+            ManifestContentType::Deletes => Ok(builder.build_v3_deletes()),
+        },
+    }
+}
+
+/// Filters existing manifests by removing entries for deleted data files.
+///
+/// This is equivalent to Java's `ManifestFilterManager`. When a rewrite or
+/// overwrite operation deletes files, the filter manager rewrites affected
+/// manifests so that deleted entries are dropped and surviving entries are
+/// re-emitted with [`ManifestStatus::Existing`].
+pub(crate) struct ManifestFilterManager {
+    deleted_file_paths: HashSet<String>,
+    fail_missing_delete_paths: bool,
+}
+
+impl ManifestFilterManager {
+    pub(crate) fn new(fail_missing_delete_paths: bool) -> Self {
+        Self {
+            deleted_file_paths: HashSet::new(),
+            fail_missing_delete_paths,
+        }
+    }
+
+    pub(crate) fn add_delete(&mut self, path: String) {
+        self.deleted_file_paths.insert(path);
+    }
+
+    /// Filter `manifests` by removing entries whose file path is in the delete
+    /// set. Returns the surviving manifests plus a [`SnapshotSummaryCollector`]
+    /// that recorded metrics for every removed file.
+    pub(crate) async fn filter_manifests(
+        &self,
+        table: &Table,
+        manifests: Vec<ManifestFile>,
+        snapshot_id: i64,
+    ) -> Result<(Vec<ManifestFile>, SnapshotSummaryCollector)> {
+        if self.deleted_file_paths.is_empty() {
+            return Ok((manifests, SnapshotSummaryCollector::default()));
+        }
+
+        let mut result: Vec<ManifestFile> = Vec::with_capacity(manifests.len());
+        let mut removed_collector = SnapshotSummaryCollector::default();
+        let mut found_paths: HashSet<String> = HashSet::new();
+
+        for manifest_file in &manifests {
+            // Only filter data manifests; pass delete manifests through unchanged.
+            if manifest_file.content != ManifestContentType::Data {
+                result.push(manifest_file.clone());
+                continue;
+            }
+
+            let manifest = table.manifest_reader().read(manifest_file).await?;
+
+            // Check whether this manifest contains any files we want to delete.
+            let has_deletes = manifest
+                .entries()
+                .iter()
+                .any(|e| e.is_alive() && self.deleted_file_paths.contains(e.file_path()));
+
+            if !has_deletes {
+                // Manifest is unaffected — pass through verbatim.
+                result.push(manifest_file.clone());
+                continue;
+            }
+
+            // Resolve the partition spec for this manifest. After partition
+            // evolution, old manifests may carry a non-default spec — we must
+            // use the manifest's own spec so the rewritten manifest stays valid.
+            let manifest_spec = table
+                .metadata()
+                .partition_spec_by_id(manifest_file.partition_spec_id)
+                .ok_or_else(|| {
+                    Error::new(
+                        ErrorKind::DataInvalid,
+                        format!(
+                            "Manifest references unknown partition spec {}",
+                            manifest_file.partition_spec_id,
+                        ),
+                    )
+                })?
+                .clone();
+            let schema = table.metadata().current_schema().clone();
+
+            // Rewrite: keep surviving entries as EXISTING, drop deleted ones.
+            let mut surviving_entries: Vec<ManifestEntry> = Vec::new();
+            for entry in manifest.entries() {
+                if entry.is_alive() && self.deleted_file_paths.contains(entry.file_path()) {
+                    // Record removal metrics.
+                    found_paths.insert(entry.file_path().to_string());
+                    removed_collector.remove_file(
+                        entry.data_file(),
+                        schema.clone(),
+                        manifest_spec.clone(),
+                    );
+                } else if entry.is_alive() {
+                    // Surviving entry — re-emit as EXISTING with original ids preserved.
+                    let existing = ManifestEntry::builder()
+                        .status(ManifestStatus::Existing)
+                        .snapshot_id(
+                            entry
+                                .snapshot_id()
+                                .unwrap_or(manifest_file.added_snapshot_id),
+                        )
+                        .sequence_number(entry.sequence_number().unwrap_or(0))
+                        .file_sequence_number(entry.file_sequence_number.unwrap_or(0))

Review Comment:
   Fixed! Both `sequence_number` and `file_sequence_number` now return explicit errors.
   
   I kept `snapshot_id` as `unwrap_or(manifest_file.added_snapshot_id)` though —
   
   V1 entries don't have per-entry snapshot_id, so the manifest-level `added_snapshot_id` is actually the right fallback there. Unlike sequence_number where 0 would just be wrong.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "hotcache (via GitHub)" <gi...@apache.org>.
hotcache commented on code in PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#discussion_r3849141307


##########
crates/iceberg/src/transaction/merging.rs:
##########
@@ -0,0 +1,398 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Shared engine for snapshot-producing operations that both add and delete files.
+//!
+//! [`MergingSnapshotProducer`] is the Rust equivalent of Java's
+//! `MergingSnapshotProducer`. It handles manifest filtering, new manifest
+//! creation, summary computation, and delegates the final snapshot commit to
+//! [`SnapshotProducer`].
+
+use std::collections::{HashMap, HashSet};
+
+use uuid::Uuid;
+
+use crate::error::Result;
+use crate::io::OutputFile;
+use crate::spec::{
+    DataContentType, DataFile, DataFileFormat, FormatVersion, ManifestContentType, ManifestEntry,
+    ManifestFile, ManifestStatus, ManifestWriter, ManifestWriterBuilder, Operation, PartitionSpec,
+    SchemaRef, SnapshotSummaryCollector, Summary, update_snapshot_summaries,
+};
+use crate::table::Table;
+use crate::transaction::ActionCommit;
+use crate::transaction::snapshot::SnapshotProducer;
+use crate::{Error, ErrorKind};
+
+/// Create a manifest writer that handles encryption when available.
+fn new_manifest_writer(
+    table: &Table,
+    output_file: OutputFile,
+    snapshot_id: Option<i64>,
+    content: ManifestContentType,
+    schema: SchemaRef,
+    partition_spec: PartitionSpec,
+) -> Result<ManifestWriter> {
+    let builder = if let Some(em) = table.encryption_manager() {
+        ManifestWriterBuilder::new_from_encrypted(
+            em.encrypt(output_file),
+            snapshot_id,
+            schema,
+            partition_spec,
+        )?
+    } else {
+        ManifestWriterBuilder::new(output_file, snapshot_id, schema, partition_spec)
+    };
+
+    match table.metadata().format_version() {
+        FormatVersion::V1 => Ok(builder.build_v1()),
+        FormatVersion::V2 => match content {
+            ManifestContentType::Data => Ok(builder.build_v2_data()),
+            ManifestContentType::Deletes => Ok(builder.build_v2_deletes()),
+        },
+        FormatVersion::V3 => match content {
+            ManifestContentType::Data => Ok(builder.build_v3_data()),
+            ManifestContentType::Deletes => Ok(builder.build_v3_deletes()),
+        },
+    }
+}
+
+/// Filters existing manifests by removing entries for deleted data files.
+///
+/// This is equivalent to Java's `ManifestFilterManager`. When a rewrite or
+/// overwrite operation deletes files, the filter manager rewrites affected
+/// manifests so that deleted entries are dropped and surviving entries are
+/// re-emitted with [`ManifestStatus::Existing`].
+pub(crate) struct ManifestFilterManager {
+    deleted_file_paths: HashSet<String>,
+    fail_missing_delete_paths: bool,
+}
+
+impl ManifestFilterManager {
+    pub(crate) fn new(fail_missing_delete_paths: bool) -> Self {
+        Self {
+            deleted_file_paths: HashSet::new(),
+            fail_missing_delete_paths,
+        }
+    }
+
+    pub(crate) fn add_delete(&mut self, path: String) {
+        self.deleted_file_paths.insert(path);
+    }
+
+    /// Filter `manifests` by removing entries whose file path is in the delete
+    /// set. Returns the surviving manifests plus a [`SnapshotSummaryCollector`]
+    /// that recorded metrics for every removed file.
+    pub(crate) async fn filter_manifests(
+        &self,
+        table: &Table,
+        manifests: Vec<ManifestFile>,
+        snapshot_id: i64,
+    ) -> Result<(Vec<ManifestFile>, SnapshotSummaryCollector)> {
+        if self.deleted_file_paths.is_empty() {
+            return Ok((manifests, SnapshotSummaryCollector::default()));
+        }
+
+        let mut result: Vec<ManifestFile> = Vec::with_capacity(manifests.len());
+        let mut removed_collector = SnapshotSummaryCollector::default();
+        let mut found_paths: HashSet<String> = HashSet::new();
+
+        for manifest_file in &manifests {
+            // Only filter data manifests; pass delete manifests through unchanged.
+            if manifest_file.content != ManifestContentType::Data {
+                result.push(manifest_file.clone());
+                continue;
+            }
+
+            let manifest = table.manifest_reader().read(manifest_file).await?;
+
+            // Check whether this manifest contains any files we want to delete.
+            let has_deletes = manifest
+                .entries()
+                .iter()
+                .any(|e| e.is_alive() && self.deleted_file_paths.contains(e.file_path()));
+
+            if !has_deletes {
+                // Manifest is unaffected — pass through verbatim.
+                result.push(manifest_file.clone());
+                continue;
+            }
+
+            // Resolve the partition spec for this manifest. After partition
+            // evolution, old manifests may carry a non-default spec — we must
+            // use the manifest's own spec so the rewritten manifest stays valid.
+            let manifest_spec = table
+                .metadata()
+                .partition_spec_by_id(manifest_file.partition_spec_id)
+                .ok_or_else(|| {
+                    Error::new(
+                        ErrorKind::DataInvalid,
+                        format!(
+                            "Manifest references unknown partition spec {}",
+                            manifest_file.partition_spec_id,
+                        ),
+                    )
+                })?
+                .clone();
+            let schema = table.metadata().current_schema().clone();
+
+            // Rewrite: keep surviving entries as EXISTING, drop deleted ones.
+            let mut surviving_entries: Vec<ManifestEntry> = Vec::new();
+            for entry in manifest.entries() {
+                if entry.is_alive() && self.deleted_file_paths.contains(entry.file_path()) {
+                    // Record removal metrics.
+                    found_paths.insert(entry.file_path().to_string());
+                    removed_collector.remove_file(
+                        entry.data_file(),
+                        schema.clone(),
+                        manifest_spec.clone(),
+                    );
+                } else if entry.is_alive() {
+                    // Surviving entry — re-emit as EXISTING with original ids preserved.
+                    let existing = ManifestEntry::builder()
+                        .status(ManifestStatus::Existing)
+                        .snapshot_id(
+                            entry
+                                .snapshot_id()
+                                .unwrap_or(manifest_file.added_snapshot_id),
+                        )
+                        .sequence_number(entry.sequence_number().unwrap_or(0))
+                        .file_sequence_number(entry.file_sequence_number.unwrap_or(0))
+                        .data_file(entry.data_file().clone())
+                        .build();
+                    surviving_entries.push(existing);
+                }
+                // Already-deleted entries (status == Deleted) are dropped.
+            }
+
+            if surviving_entries.is_empty() {
+                // Manifest is now empty — omit entirely.
+                continue;
+            }
+
+            // Write the filtered manifest using the manifest's own partition spec.
+            let new_manifest_path = format!(
+                "{}/{}-m-filter-{}.{}",
+                table.metadata().metadata_location()?,

Review Comment:
   Fixed! Now using `<uuid>-m<counter>.avro` like SnapshotProducer does.
   
   The counter is a local `RangeFrom<u64>` shared across `filter_manifests` and `write_added_manifest` within each `commit_snapshot` call.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#discussion_r4192688698


##########
crates/iceberg/src/transaction/rewrite.rs:
##########
@@ -0,0 +1,1353 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transaction action for rewriting data files (compaction).
+//!
+//! [`RewriteFilesAction`] replaces a set of data files with a new set while
+//! keeping the logical table contents unchanged. This is used for compaction —
+//! merging many small files into fewer large ones.
+//!
+//! The resulting snapshot uses [`Operation::Replace`] to indicate that files
+//! were reorganised without changing the data.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+
+use crate::error::{Result, invalid_data};
+use crate::spec::{DataFile, ManifestContentType, Operation};
+use crate::table::Table;
+use crate::transaction::merging::MergingSnapshotProducer;
+use crate::transaction::{ActionCommit, TransactionAction};
+
+/// A transaction action that rewrites (replaces) data files.
+///
+/// This is the Rust equivalent of Java's `BaseRewriteFiles`. It uses
+/// `MergingSnapshotProducer` to handle manifest filtering and creation,
+/// and commits a snapshot with [`Operation::Replace`].
+///
+/// # Example
+///
+/// ```ignore
+/// let tx = Transaction::new(&table);
+/// let action = tx.rewrite_files()
+///     .delete_file(old_file_1)
+///     .delete_file(old_file_2)
+///     .add_file(merged_file);
+/// let tx = action.apply(tx)?;
+/// let table = tx.commit(&catalog).await?;
+/// ```
+///
+/// # Concurrent deletes
+///
+/// The action records the table's current snapshot when it is created, and
+/// refuses to commit if the table gained a delete manifest after it. Java
+/// narrows that to the files being replaced in
+/// `validateNoNewDeletesForDataFiles`; until that exists here the check is
+/// table-wide, so a delete committed to an unrelated partition while the
+/// rewrite ran also fails the commit with
+/// [`ErrorKind::DataInvalid`](crate::ErrorKind::DataInvalid). So do a
+/// rewrite planned against a table that had no snapshot yet, and one whose
+/// starting snapshot has since been expired: neither can rule out a delete.
+/// Replan the rewrite against the current table and run it again.
+///
+/// Deletes that were already committed when the rewrite was planned are not
+/// covered by this check. Either apply them while rewriting, or keep them
+/// applicable to the new file with
+/// [`data_sequence_number`](RewriteFilesAction::data_sequence_number).
+pub struct RewriteFilesAction {
+    producer: MergingSnapshotProducer,
+    /// The snapshot the rewrite was planned against, if the table had one.
+    /// Deletes newer than it fail the commit.
+    starting_snapshot_id: Option<i64>,
+}
+
+impl RewriteFilesAction {
+    pub(crate) fn new(starting_snapshot_id: Option<i64>) -> Self {
+        Self {
+            producer: MergingSnapshotProducer::new(Operation::Replace),
+            starting_snapshot_id,
+        }
+    }
+
+    /// Register a data file to be removed from the table.
+    ///
+    /// The file must exist in the current snapshot; otherwise the commit
+    /// will fail with a validation error.
+    pub fn delete_file(mut self, file: DataFile) -> Self {
+        self.producer.delete_data_file(file);
+        self
+    }
+
+    /// Register a data file to be added to the table.
+    ///
+    /// Typically this is the merged output of the files being deleted.
+    pub fn add_file(mut self, file: DataFile) -> Self {
+        self.producer.add_data_file(file);
+        self
+    }
+
+    /// Set the data sequence number recorded for every added file.
+    ///
+    /// Without this the added files inherit the sequence number of the new
+    /// snapshot. A compaction that must not shadow concurrently written
+    /// deletes sets the sequence number of the files it replaces instead.
+    /// V1 manifest entries carry no sequence number, so this has no effect on
+    /// a V1 table.
+    pub fn data_sequence_number(mut self, sequence_number: i64) -> Self {
+        self.producer.set_data_sequence_number(sequence_number);
+        self
+    }
+
+    fn validate(&self) -> Result<()> {
+        if !self.producer.has_deleted_data_files() {
+            return Err(invalid_data!(
+                "Rewrite files requires at least one file to delete"
+            ));
+        }
+        if !self.producer.has_added_data_files() {
+            return Err(invalid_data!(
+                "Rewrite files requires at least one file to add"
+            ));
+        }

Review Comment:
   Should this follow `BaseRewriteFiles` and allow a rewrite with no added files? Java's `validateReplacedAndAddedFiles` only requires the removal side to be non-empty ("Files to delete cannot be empty"); its other two checks forbid adds without matching removals, not the reverse.
   
   Remove-only rewrites are legitimate even at this PR's data-file-only scope: a compaction group whose remaining rows are all shadowed by deletes produces zero output files, and this check would reject that commit. It also matters for the follow-ups — `RemoveDanglingDeleteFiles` is exactly a remove-only `RewriteFiles`, and rewritten position-delete groups can shrink to nothing once dangling rows are dropped.
   
   Concretely: drop this check and keep only the `has_deleted_data_files()` requirement above (which then generalizes to "deletes data or delete files" once delete-file removal lands). Happy to send that as part of the delete-file follow-up instead if you'd rather not touch it here — but relaxing a validation after release is noisier than merging the Java-parity rule now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "rexminnis (via GitHub)" <gi...@apache.org>.
rexminnis commented on PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#issuecomment-5874670479

   
   As promised, a run against a real REST catalog: Apache Polaris 1.7.0 with S3 and vended credentials, on this branch plus hotcache#2.
   
   **Setup.** The table was written by a different implementation. PyIceberg 0.12 created a format-v2 table partitioned by `identity(ts)` and appended three files, one per distinct `ts`. It then evolved the spec to `day(ts)` and appended one more file. That gives 400 rows, three spec-0 files in one manifest, and one spec-1 file. The rewrites use `RewriteFilesAction` from this branch. Trino 483 (Java) reads the result independently, and PyIceberg cross-checks it.
   
   1. **Rewrite 1:** two of the three spec-0 files are compacted into one `day(ts)` file.
      - The survivor is re-emitted `Existing` in a rewritten **spec-0** manifest. It keeps its original snapshot id, `seq=1`, `file_seq=1` and timestamptz partition value, and the manifest's `min_sequence_number` is 1.
      - Trino's `$files` still shows the survivor as `spec_id 0` with `ts=2026-08-24 17:00:00 UTC`.
      - Totals are exact: 3 files and 400 rows.
      - `changed-partition-count=3`, with `partitions.ts=…` for the two removed files and `partitions.ts_day=2026-08-24` for the added one. That summary is the hotcache#2 fix, now live.
   2. **Rewrite 2:** the spec-0 survivor and the PyIceberg spec-1 file are compacted into one file.
      - Both emptied manifests are dropped, and no spec-0 file or manifest remains.
      - Totals chain exactly through both replaces: 400 rows and 2 files, and `total-files-size` equals the two live files.
   3. **Independent reads:** Trino gives the same answer at every step, and so does time travel to each earlier snapshot. PyIceberg matches on the final snapshot: `count=400`, `sum(id)=79800`, 400 distinct ids and `sum(v)=39900`.
   
   **One gap the live run showed: removed files are not written as `DELETED` entries.** `filter_manifests` drops the removed entries. Java's `ManifestFilterManager` writes each one with `writer.delete(entry)`, an entry with status `DELETED` owned by the removing snapshot. On this table, Trino's `$manifests` reports `deleted_data_files_count = 0` for every manifest, while the snapshot summary says `deleted-data-files=2`.
   
   The practical consequence is in snapshot expiration. For a linear, main-branch-only history, `RemoveSnapshots` picks `IncrementalFileCleanup`, which finds data files to delete **only** through `DELETED` entries whose snapshot has expired (`IncrementalFileCleanup#findFilesToDelete`). So a Java `expireSnapshots` run on a table compacted by this action would never delete the replaced data files; they'd stay in storage as orphans. Changelog readers that walk manifest entries would miss the removals too.
   
   `ManifestWriter::add_delete_entry` already exists, so writing the removed entries as `DELETED` with the new snapshot id looks like a small change. It does change one rule, though: a manifest whose entries were all removed by the current snapshot still has to be written, holding only `DELETED` entries. The next commit then drops it, as `ee3fbdd` already does for manifests left all-deleted by an earlier snapshot. That matches Java, which keeps a manifest if `snapshotId() == current`. `test_rewrite_files_removes_empty_manifest` and the `retires_the_old_spec` fixture in hotcache#2 assert that the manifest is absent. They would need to assert that it has no live entries instead. I'm happy to adjust the fixture if you take this.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "jopdorp (via GitHub)" <gi...@apache.org>.
jopdorp commented on code in PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#discussion_r4258448054


##########
crates/iceberg/src/transaction/rewrite.rs:
##########
@@ -0,0 +1,1353 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Transaction action for rewriting data files (compaction).
+//!
+//! [`RewriteFilesAction`] replaces a set of data files with a new set while
+//! keeping the logical table contents unchanged. This is used for compaction —
+//! merging many small files into fewer large ones.
+//!
+//! The resulting snapshot uses [`Operation::Replace`] to indicate that files
+//! were reorganised without changing the data.
+
+use std::sync::Arc;
+
+use async_trait::async_trait;
+
+use crate::error::{Result, invalid_data};
+use crate::spec::{DataFile, ManifestContentType, Operation};
+use crate::table::Table;
+use crate::transaction::merging::MergingSnapshotProducer;
+use crate::transaction::{ActionCommit, TransactionAction};
+
+/// A transaction action that rewrites (replaces) data files.
+///
+/// This is the Rust equivalent of Java's `BaseRewriteFiles`. It uses
+/// `MergingSnapshotProducer` to handle manifest filtering and creation,
+/// and commits a snapshot with [`Operation::Replace`].
+///
+/// # Example
+///
+/// ```ignore
+/// let tx = Transaction::new(&table);
+/// let action = tx.rewrite_files()
+///     .delete_file(old_file_1)
+///     .delete_file(old_file_2)
+///     .add_file(merged_file);
+/// let tx = action.apply(tx)?;
+/// let table = tx.commit(&catalog).await?;
+/// ```
+///
+/// # Concurrent deletes
+///
+/// The action records the table's current snapshot when it is created, and
+/// refuses to commit if the table gained a delete manifest after it. Java
+/// narrows that to the files being replaced in
+/// `validateNoNewDeletesForDataFiles`; until that exists here the check is
+/// table-wide, so a delete committed to an unrelated partition while the
+/// rewrite ran also fails the commit with
+/// [`ErrorKind::DataInvalid`](crate::ErrorKind::DataInvalid). So do a
+/// rewrite planned against a table that had no snapshot yet, and one whose
+/// starting snapshot has since been expired: neither can rule out a delete.
+/// Replan the rewrite against the current table and run it again.
+///
+/// Deletes that were already committed when the rewrite was planned are not
+/// covered by this check. Either apply them while rewriting, or keep them
+/// applicable to the new file with
+/// [`data_sequence_number`](RewriteFilesAction::data_sequence_number).
+pub struct RewriteFilesAction {
+    producer: MergingSnapshotProducer,
+    /// The snapshot the rewrite was planned against, if the table had one.
+    /// Deletes newer than it fail the commit.
+    starting_snapshot_id: Option<i64>,
+}
+
+impl RewriteFilesAction {
+    pub(crate) fn new(starting_snapshot_id: Option<i64>) -> Self {
+        Self {
+            producer: MergingSnapshotProducer::new(Operation::Replace),
+            starting_snapshot_id,
+        }
+    }
+
+    /// Register a data file to be removed from the table.
+    ///
+    /// The file must exist in the current snapshot; otherwise the commit
+    /// will fail with a validation error.
+    pub fn delete_file(mut self, file: DataFile) -> Self {
+        self.producer.delete_data_file(file);
+        self
+    }
+
+    /// Register a data file to be added to the table.
+    ///
+    /// Typically this is the merged output of the files being deleted.
+    pub fn add_file(mut self, file: DataFile) -> Self {
+        self.producer.add_data_file(file);
+        self
+    }
+
+    /// Set the data sequence number recorded for every added file.
+    ///
+    /// Without this the added files inherit the sequence number of the new
+    /// snapshot. A compaction that must not shadow concurrently written
+    /// deletes sets the sequence number of the files it replaces instead.
+    /// V1 manifest entries carry no sequence number, so this has no effect on
+    /// a V1 table.
+    pub fn data_sequence_number(mut self, sequence_number: i64) -> Self {
+        self.producer.set_data_sequence_number(sequence_number);
+        self
+    }
+
+    fn validate(&self) -> Result<()> {
+        if !self.producer.has_deleted_data_files() {
+            return Err(invalid_data!(
+                "Rewrite files requires at least one file to delete"
+            ));
+        }
+        if !self.producer.has_added_data_files() {
+            return Err(invalid_data!(
+                "Rewrite files requires at least one file to add"
+            ));
+        }

Review Comment:
   sent this as https://github.com/hotcache/iceberg-rust/pull/3. remove-only rewrites now keep the tombstone and correct totals; the library suite passes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "rexminnis (via GitHub)" <gi...@apache.org>.
rexminnis commented on PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#issuecomment-5874223407

   
   Thanks for working through all of it, and thanks @jopdorp for the fixes.
   
   The spec-evolution fixture is up as hotcache/iceberg-rust#2, two commits on top of `e7db3c8`:
   
   - **`test(transaction): cover rewrites on a spec-evolved table`**: a table partitioned by `identity(ts)`, then evolved to `day(ts)`, run on V2 and V3. A survivor stays `Existing` in a spec-0 manifest, with its timestamptz partition value and that manifest's partition summary unchanged. One rewrite filters one manifest of each spec. Retiring the last old-spec file drops the spec-0 manifest, and the totals stay exact across both rewrites. All three pass on your branch as it is, so the spec lookup in the filter holds up.
   - **`fix(transaction): keep partition metrics in the rewrite summary`**: the fixture found one gap. `build_summary` merges the filter's removal collector into a new one, and `SnapshotSummaryCollector::merge` drops partition metrics unless both sides trust them. Neither side does, because `Default` leaves `trust_partition_metrics` false. The result is that a replace snapshot never reports `changed-partition-count` or `partitions.*`. A fast append on the same table reports `changed-partition-count=1`, and the rewrite reports nothing. The fix adds the new files to the filter's collector instead of merging, and honours `write.summary.partition-limit` the same way `SnapshotProducer::summary` does. With the fix, the rewrite reports `ts=2026-08-24+09%3A30%3A00+UTC` for the removed file and `ts_day=2026-08-24` for the added one, each named under its own spec. The new test fails without the fix.
   
   (Java's `SnapshotSummary.Builder` starts with `trustPartitionMetrics = true`. The Rust default is the other way round, and `snapshot_summary.rs` has a test that asserts merging two default collectors drops the partition summaries. So I kept the fix local to the rewrite rather than changing the shared default. That default may deserve its own issue.)
   
   `cargo test -p iceberg --lib` passes 1817 tests, and `cargo fmt --check` and `clippy -D warnings` are clean. No public API changes.
   
   The REST-catalog run with an independent reader is next. I'll report back here once it's done.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "jopdorp (via GitHub)" <gi...@apache.org>.
jopdorp commented on PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#issuecomment-5733128922

   We need compaction and, on top of this producer, a row-delta action for a service we build on iceberg-rust with a Lakekeeper catalog, so I worked through the review and did the fixes on top of your branch: hotcache/iceberg-rust#1. One commit per point from @rexminnis and the V2 tests @JanKaul asked for, each with a test that fails on the current code. One choice to flag: the manifest counter lives on the action rather than the producer, because the producer is rebuilt on every commit attempt and the action is what survives a retry. Happy to adjust anything.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "hotcache (via GitHub)" <gi...@apache.org>.
hotcache commented on PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#issuecomment-5870613587

   Rebased on main and worked through the review.
   
   @jopdorp did most of it in `hotcache/iceberg-rust#1` — one commit per point, each
   with a test that fails on the code before it. His commits are here with their
   authorship; when this is squashed, please keep:
   
       Co-authored-by: Jegor van Opdorp <je...@gmail.com>
   
   **@JanKaul** — both inline comments are fixed in `7fdc8fd`. The compaction and
   partial-rewrite tests now run against V2 (`6fcdb66`) and V1 (`69aa1b3`) as well,
   sharing the same read-back assertions.
   
   **@rexminnis** — (1) survivors go through `add_existing_entry` and keep their
   status, snapshot id and both sequence numbers (`83ac7a5`); (2)
   `data_sequence_number()`, bounded by the new snapshot's (`6fc7bf1`); (4) both
   producer validations run before a rewrite commits (`9d40b4c`); (5) the counter
   moved onto `MergingSnapshotProducer`, which the action owns across retries, so
   attempt 2 cannot overwrite attempt 1's manifest (`d466a48`); (6) all-deleted
   manifests drop out (`ee3fbdd`). (3) is interim — see below.
   
   Two notes. On (1), `add_entry` also left the entry's sequence number `None`, so
   `ManifestWriter` never updated `min_seq_num` and the manifest list stamped the
   rewritten manifest with the *new* snapshot's `min_sequence_number` — delete-file
   pruning was reading a wrong lower bound. On (5) and (6), `fast_append` has the
   same two shapes and is untouched here (#2545 for the latter).
   
   ### The interim conflict check
   
   `4f26afc` refuses a rewrite when a delete manifest landed after its starting
   snapshot. It was fail-*open* in two cases, both of which committed silently;
   each now has a test that fails without the fix:
   
   - the starting snapshot has since been expired, where the comparison fell back
     to sequence number 0 (`d22436b`);
   - the rewrite was planned before the table had any snapshot, where the check was
     skipped although *everything* the table holds arrived after planning
     (`f5818cc`).
   
   `d4dbf46` documents the resulting behaviour: the check is table-wide and always
   on, so a delete committed anywhere while the compaction ran fails the commit.
   That is the intended trade until the file-level check exists, but a caller has to
   be able to tell it from a bug.
   
   Also `b8c4c70` — `validate_added_data_files` is shared with fast append, so its
   content-type error claimed "for fast append" and did not name the file.
   
   ### On the design
   
   RFC 0003 landed in the meantime (#2620). §4.1 specifies independent concrete
   producer types rather than an inheritance hierarchy, with the producer retaining
   work across retries — which is what this PR does and what (5) relies on. The RFC
   lists `RowDeltaAction` as the initial consumer of `MergingSnapshotProducer`;
   RewriteFiles arriving first doesn't change the shape, and RowDelta is PR5 here.
   
   `e7db3c8` also moves the new code onto the `invalid_data!` macro from #2928,
   which landed on main while this was open.
   
   ### Not in this PR
   
   - **File-level `validateNoNewDeletesForDataFiles`** — needs the snapshot
     validation in #2243; PR6 in the series. The table-wide refusal stands in.
   - **Partition-summary pruning** (Java's `canContainDeletedFiles`) — every data
     manifest is read today.
   - **`set_commit_uuid` and snapshot properties on the action** — asymmetric with
     `fast_append`.
   - **A second partition spec in the tests.** The spec lookup in the filter is
     still only exercised by the default spec, and it is the part that matters most
     on an evolved table. @rexminnis — you offered an `identity(ts)` → `day(ts)`
     fixture and an independent reader against a REST catalog once (1) and (2) were
     in; they are, so that would be very welcome, here or as a follow-up.
   
   `cargo test -p iceberg`, `cargo fmt`, `clippy -D warnings` and
   `make check-public-api` are clean; `transaction::rewrite` has 21 tests.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "rexminnis (via GitHub)" <gi...@apache.org>.
rexminnis commented on PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#issuecomment-5711613343

   Thanks for pushing this forward — the filter manager resolving each manifest's own partition spec and the summary subtracting removed files (with the `total-*` assertions in the tests) are both things this operation has to get right, and it's good to see them here. A few findings from reading the committed manifests back rather than only the summary:
   
   **1. Surviving entries are written as `Added`, not `Existing` (blocking).**
   `filter_manifests` builds each survivor as `Existing` with the original snapshot id and sequence numbers, then passes it to `ManifestWriter::add_entry`. Per that method's doc it resets status to `Added`, snapshot id to the writer's, and `file_sequence_number` to `None`. A read-back test on this branch (append two files, rewrite one, then read the manifests of the replace snapshot) shows the survivor as `status=Added`, `snapshot_id=<replace snapshot>`, `sequence_number=1`, `file_sequence_number=2`, and the filtered `ManifestFile` reports `added_files_count=1, existing_files_count=0`. Incremental readers will re-emit the survivor as new data on every compaction that touches its manifest. `ManifestWriter::add_existing_entry` (currently `#[allow(dead_code)]`) preserves all three fields; with that one-line swap the same test passes with `status=Existing`, the append snapshot's id, `sequence_number=1`, `file_sequence_number=1`, and `added=0, existing=1`, and the existing seven tests sti
 ll pass. Worth adding that read-back assertion to `test_rewrite_files_partial` — `is_alive()` alone can't distinguish the two.
   
   **2. Added files need a data sequence number option.**
   Added entries inherit the new snapshot's sequence number. On a table with equality deletes, compacting files at sequence 3 and 4 into a file at sequence 9 means a delete at sequence 5 no longer applies to the new file, so rows it removed come back unless the rewriter applied it. Java exposes `RewriteFiles.dataSequenceNumber(long)` for this (Spark's rewrite sets it to the starting snapshot's sequence by default). `ManifestWriter::add_file(data_file, sequence_number)` already exists, so this is mostly threading a value through `write_added_manifest`. Without it the action should probably document that all delete files must have been applied.
   
   **3. Conflict validation / `starting_snapshot_id`.**
   Understood this is the follow-up PR. One note on the interim: `Transaction::do_commit` re-runs `commit` against the refreshed table on retry, so the filter silently re-bases onto whatever is current, and a position or equality delete committed concurrently against a replaced file is dropped. Until `validateNoNewDeletesForDataFiles` lands, refusing when `starting_snapshot_id` is set and any delete manifest was added after it would keep this fail-closed. (The missing-source-file case is already fail-closed: `DataInvalid` isn't retryable.)
   
   **4. Producer validations are bypassed.**
   `commit_with_manifests` skips `validate_added_data_files` (the `partition_spec_id == default` and `validate_partition_value` checks — only the content-type half is repeated here) and `validate_duplicate_files`. Adding a file whose partition struct doesn't match the default spec writes a malformed manifest, and adding an already-live path creates a duplicate entry. Running both in `commit_snapshot` closes it.
   
   **5. Manifest paths across retries.**
   `commit_uuid` is fixed per action and `manifest_counter` restarts at 0 on every `commit_snapshot` call, so attempt 2 writes `{uuid}-m0.avro` over attempt 1's. That's fine when attempt 1 provably failed, but after an ambiguous catalog outcome it overwrites a manifest the live snapshot may reference — and here that manifest holds surviving entries, not only the new file. Keeping the counter on the producer (Java never resets `manifestCount`) or including the attempt in the path avoids it. `fast_append` has the same shape, so this may be worth fixing in `SnapshotProducer` instead.
   
   **6. All-deleted manifests are kept forever.**
   The manifest load keeps entries with `has_deleted_files()`. Java's filter keeps `hasAddedFiles() || hasExistingFiles() || snapshotId() == current`, so manifests that went all-deleted in an earlier commit drop out; here they ride along in every future manifest list.
   
   Follow-ups, not blocking: no partition-summary pruning before reading every data manifest (Java's `canContainDeletedFiles`); no snapshot properties / `set_commit_uuid` on the action, unlike `fast_append`; and none of the tests exercise a second partition spec, so the spec lookup in the filter (the part that matters most on evolved tables) is untested. Happy to contribute a spec-evolution fixture (`identity(ts)` → `day(ts)`) and to run this against a REST catalog with an independent reader once 1 and 2 are in.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#issuecomment-6453377036

   @jopdorp  -- i'm fine with that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "JanKaul (via GitHub)" <gi...@apache.org>.
JanKaul commented on code in PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#discussion_r3842337390


##########
crates/iceberg/src/transaction/merging.rs:
##########
@@ -0,0 +1,398 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Shared engine for snapshot-producing operations that both add and delete files.
+//!
+//! [`MergingSnapshotProducer`] is the Rust equivalent of Java's
+//! `MergingSnapshotProducer`. It handles manifest filtering, new manifest
+//! creation, summary computation, and delegates the final snapshot commit to
+//! [`SnapshotProducer`].
+
+use std::collections::{HashMap, HashSet};
+
+use uuid::Uuid;
+
+use crate::error::Result;
+use crate::io::OutputFile;
+use crate::spec::{
+    DataContentType, DataFile, DataFileFormat, FormatVersion, ManifestContentType, ManifestEntry,
+    ManifestFile, ManifestStatus, ManifestWriter, ManifestWriterBuilder, Operation, PartitionSpec,
+    SchemaRef, SnapshotSummaryCollector, Summary, update_snapshot_summaries,
+};
+use crate::table::Table;
+use crate::transaction::ActionCommit;
+use crate::transaction::snapshot::SnapshotProducer;
+use crate::{Error, ErrorKind};
+
+/// Create a manifest writer that handles encryption when available.
+fn new_manifest_writer(
+    table: &Table,
+    output_file: OutputFile,
+    snapshot_id: Option<i64>,
+    content: ManifestContentType,
+    schema: SchemaRef,
+    partition_spec: PartitionSpec,
+) -> Result<ManifestWriter> {
+    let builder = if let Some(em) = table.encryption_manager() {
+        ManifestWriterBuilder::new_from_encrypted(
+            em.encrypt(output_file),
+            snapshot_id,
+            schema,
+            partition_spec,
+        )?
+    } else {
+        ManifestWriterBuilder::new(output_file, snapshot_id, schema, partition_spec)
+    };
+
+    match table.metadata().format_version() {
+        FormatVersion::V1 => Ok(builder.build_v1()),
+        FormatVersion::V2 => match content {
+            ManifestContentType::Data => Ok(builder.build_v2_data()),
+            ManifestContentType::Deletes => Ok(builder.build_v2_deletes()),
+        },
+        FormatVersion::V3 => match content {
+            ManifestContentType::Data => Ok(builder.build_v3_data()),
+            ManifestContentType::Deletes => Ok(builder.build_v3_deletes()),
+        },
+    }
+}
+
+/// Filters existing manifests by removing entries for deleted data files.
+///
+/// This is equivalent to Java's `ManifestFilterManager`. When a rewrite or
+/// overwrite operation deletes files, the filter manager rewrites affected
+/// manifests so that deleted entries are dropped and surviving entries are
+/// re-emitted with [`ManifestStatus::Existing`].
+pub(crate) struct ManifestFilterManager {
+    deleted_file_paths: HashSet<String>,
+    fail_missing_delete_paths: bool,
+}
+
+impl ManifestFilterManager {
+    pub(crate) fn new(fail_missing_delete_paths: bool) -> Self {
+        Self {
+            deleted_file_paths: HashSet::new(),
+            fail_missing_delete_paths,
+        }
+    }
+
+    pub(crate) fn add_delete(&mut self, path: String) {
+        self.deleted_file_paths.insert(path);
+    }
+
+    /// Filter `manifests` by removing entries whose file path is in the delete
+    /// set. Returns the surviving manifests plus a [`SnapshotSummaryCollector`]
+    /// that recorded metrics for every removed file.
+    pub(crate) async fn filter_manifests(
+        &self,
+        table: &Table,
+        manifests: Vec<ManifestFile>,
+        snapshot_id: i64,
+    ) -> Result<(Vec<ManifestFile>, SnapshotSummaryCollector)> {
+        if self.deleted_file_paths.is_empty() {
+            return Ok((manifests, SnapshotSummaryCollector::default()));
+        }
+
+        let mut result: Vec<ManifestFile> = Vec::with_capacity(manifests.len());
+        let mut removed_collector = SnapshotSummaryCollector::default();
+        let mut found_paths: HashSet<String> = HashSet::new();
+
+        for manifest_file in &manifests {
+            // Only filter data manifests; pass delete manifests through unchanged.
+            if manifest_file.content != ManifestContentType::Data {
+                result.push(manifest_file.clone());
+                continue;
+            }
+
+            let manifest = table.manifest_reader().read(manifest_file).await?;
+
+            // Check whether this manifest contains any files we want to delete.
+            let has_deletes = manifest
+                .entries()
+                .iter()
+                .any(|e| e.is_alive() && self.deleted_file_paths.contains(e.file_path()));
+
+            if !has_deletes {
+                // Manifest is unaffected — pass through verbatim.
+                result.push(manifest_file.clone());
+                continue;
+            }
+
+            // Resolve the partition spec for this manifest. After partition
+            // evolution, old manifests may carry a non-default spec — we must
+            // use the manifest's own spec so the rewritten manifest stays valid.
+            let manifest_spec = table
+                .metadata()
+                .partition_spec_by_id(manifest_file.partition_spec_id)
+                .ok_or_else(|| {
+                    Error::new(
+                        ErrorKind::DataInvalid,
+                        format!(
+                            "Manifest references unknown partition spec {}",
+                            manifest_file.partition_spec_id,
+                        ),
+                    )
+                })?
+                .clone();
+            let schema = table.metadata().current_schema().clone();
+
+            // Rewrite: keep surviving entries as EXISTING, drop deleted ones.
+            let mut surviving_entries: Vec<ManifestEntry> = Vec::new();
+            for entry in manifest.entries() {
+                if entry.is_alive() && self.deleted_file_paths.contains(entry.file_path()) {
+                    // Record removal metrics.
+                    found_paths.insert(entry.file_path().to_string());
+                    removed_collector.remove_file(
+                        entry.data_file(),
+                        schema.clone(),
+                        manifest_spec.clone(),
+                    );
+                } else if entry.is_alive() {
+                    // Surviving entry — re-emit as EXISTING with original ids preserved.
+                    let existing = ManifestEntry::builder()
+                        .status(ManifestStatus::Existing)
+                        .snapshot_id(
+                            entry
+                                .snapshot_id()
+                                .unwrap_or(manifest_file.added_snapshot_id),
+                        )
+                        .sequence_number(entry.sequence_number().unwrap_or(0))
+                        .file_sequence_number(entry.file_sequence_number.unwrap_or(0))
+                        .data_file(entry.data_file().clone())
+                        .build();
+                    surviving_entries.push(existing);
+                }
+                // Already-deleted entries (status == Deleted) are dropped.
+            }
+
+            if surviving_entries.is_empty() {
+                // Manifest is now empty — omit entirely.
+                continue;
+            }
+
+            // Write the filtered manifest using the manifest's own partition spec.
+            let new_manifest_path = format!(
+                "{}/{}-m-filter-{}.{}",
+                table.metadata().metadata_location()?,

Review Comment:
   The manifest naming here deviates from the standard naming scheme `<commit-uuid>-m<counter>.avro`. I personally would stay with the standard scheme and don't invent any new concepts. Technically it would be fine.



##########
crates/iceberg/src/transaction/merging.rs:
##########
@@ -0,0 +1,398 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Shared engine for snapshot-producing operations that both add and delete files.
+//!
+//! [`MergingSnapshotProducer`] is the Rust equivalent of Java's
+//! `MergingSnapshotProducer`. It handles manifest filtering, new manifest
+//! creation, summary computation, and delegates the final snapshot commit to
+//! [`SnapshotProducer`].
+
+use std::collections::{HashMap, HashSet};
+
+use uuid::Uuid;
+
+use crate::error::Result;
+use crate::io::OutputFile;
+use crate::spec::{
+    DataContentType, DataFile, DataFileFormat, FormatVersion, ManifestContentType, ManifestEntry,
+    ManifestFile, ManifestStatus, ManifestWriter, ManifestWriterBuilder, Operation, PartitionSpec,
+    SchemaRef, SnapshotSummaryCollector, Summary, update_snapshot_summaries,
+};
+use crate::table::Table;
+use crate::transaction::ActionCommit;
+use crate::transaction::snapshot::SnapshotProducer;
+use crate::{Error, ErrorKind};
+
+/// Create a manifest writer that handles encryption when available.
+fn new_manifest_writer(
+    table: &Table,
+    output_file: OutputFile,
+    snapshot_id: Option<i64>,
+    content: ManifestContentType,
+    schema: SchemaRef,
+    partition_spec: PartitionSpec,
+) -> Result<ManifestWriter> {
+    let builder = if let Some(em) = table.encryption_manager() {
+        ManifestWriterBuilder::new_from_encrypted(
+            em.encrypt(output_file),
+            snapshot_id,
+            schema,
+            partition_spec,
+        )?
+    } else {
+        ManifestWriterBuilder::new(output_file, snapshot_id, schema, partition_spec)
+    };
+
+    match table.metadata().format_version() {
+        FormatVersion::V1 => Ok(builder.build_v1()),
+        FormatVersion::V2 => match content {
+            ManifestContentType::Data => Ok(builder.build_v2_data()),
+            ManifestContentType::Deletes => Ok(builder.build_v2_deletes()),
+        },
+        FormatVersion::V3 => match content {
+            ManifestContentType::Data => Ok(builder.build_v3_data()),
+            ManifestContentType::Deletes => Ok(builder.build_v3_deletes()),
+        },
+    }
+}
+
+/// Filters existing manifests by removing entries for deleted data files.
+///
+/// This is equivalent to Java's `ManifestFilterManager`. When a rewrite or
+/// overwrite operation deletes files, the filter manager rewrites affected
+/// manifests so that deleted entries are dropped and surviving entries are
+/// re-emitted with [`ManifestStatus::Existing`].
+pub(crate) struct ManifestFilterManager {
+    deleted_file_paths: HashSet<String>,
+    fail_missing_delete_paths: bool,
+}
+
+impl ManifestFilterManager {
+    pub(crate) fn new(fail_missing_delete_paths: bool) -> Self {
+        Self {
+            deleted_file_paths: HashSet::new(),
+            fail_missing_delete_paths,
+        }
+    }
+
+    pub(crate) fn add_delete(&mut self, path: String) {
+        self.deleted_file_paths.insert(path);
+    }
+
+    /// Filter `manifests` by removing entries whose file path is in the delete
+    /// set. Returns the surviving manifests plus a [`SnapshotSummaryCollector`]
+    /// that recorded metrics for every removed file.
+    pub(crate) async fn filter_manifests(
+        &self,
+        table: &Table,
+        manifests: Vec<ManifestFile>,
+        snapshot_id: i64,
+    ) -> Result<(Vec<ManifestFile>, SnapshotSummaryCollector)> {
+        if self.deleted_file_paths.is_empty() {
+            return Ok((manifests, SnapshotSummaryCollector::default()));
+        }
+
+        let mut result: Vec<ManifestFile> = Vec::with_capacity(manifests.len());
+        let mut removed_collector = SnapshotSummaryCollector::default();
+        let mut found_paths: HashSet<String> = HashSet::new();
+
+        for manifest_file in &manifests {
+            // Only filter data manifests; pass delete manifests through unchanged.
+            if manifest_file.content != ManifestContentType::Data {
+                result.push(manifest_file.clone());
+                continue;
+            }
+
+            let manifest = table.manifest_reader().read(manifest_file).await?;
+
+            // Check whether this manifest contains any files we want to delete.
+            let has_deletes = manifest
+                .entries()
+                .iter()
+                .any(|e| e.is_alive() && self.deleted_file_paths.contains(e.file_path()));
+
+            if !has_deletes {
+                // Manifest is unaffected — pass through verbatim.
+                result.push(manifest_file.clone());
+                continue;
+            }
+
+            // Resolve the partition spec for this manifest. After partition
+            // evolution, old manifests may carry a non-default spec — we must
+            // use the manifest's own spec so the rewritten manifest stays valid.
+            let manifest_spec = table
+                .metadata()
+                .partition_spec_by_id(manifest_file.partition_spec_id)
+                .ok_or_else(|| {
+                    Error::new(
+                        ErrorKind::DataInvalid,
+                        format!(
+                            "Manifest references unknown partition spec {}",
+                            manifest_file.partition_spec_id,
+                        ),
+                    )
+                })?
+                .clone();
+            let schema = table.metadata().current_schema().clone();
+
+            // Rewrite: keep surviving entries as EXISTING, drop deleted ones.
+            let mut surviving_entries: Vec<ManifestEntry> = Vec::new();
+            for entry in manifest.entries() {
+                if entry.is_alive() && self.deleted_file_paths.contains(entry.file_path()) {
+                    // Record removal metrics.
+                    found_paths.insert(entry.file_path().to_string());
+                    removed_collector.remove_file(
+                        entry.data_file(),
+                        schema.clone(),
+                        manifest_spec.clone(),
+                    );
+                } else if entry.is_alive() {
+                    // Surviving entry — re-emit as EXISTING with original ids preserved.
+                    let existing = ManifestEntry::builder()
+                        .status(ManifestStatus::Existing)
+                        .snapshot_id(
+                            entry
+                                .snapshot_id()
+                                .unwrap_or(manifest_file.added_snapshot_id),
+                        )
+                        .sequence_number(entry.sequence_number().unwrap_or(0))
+                        .file_sequence_number(entry.file_sequence_number.unwrap_or(0))

Review Comment:
   I think the sequence_number here shouldn't silently be set to 0 here. True, this should always be a `Some()` here but it's still better to throw an explicit error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "hotcache (via GitHub)" <gi...@apache.org>.
hotcache commented on PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#issuecomment-5892867580

   Thanks @rexminnis — both of your commits are in, and I took the `DELETED` entry change as well.
   
   **Merged from hotcache/iceberg-rust#2**, with your authorship:
   
   * `43a09a5` — the spec-evolution fixture. It passes as-is, so the filter resolving each manifest against its own spec works fine on an evolved table.
   * `89786a8` — the partition-metrics fix. `merge` dropping the metrics unless both sides trust them is a bit of a sharp edge. Keeping the fix local to the rewrite instead of changing the default in `snapshot_summary.rs` seems right to me. I added the shared default to the follow-ups in the description so it doesn't get forgotten.
   
   I closed the PR by hand since GitHub didn't close it automatically. The 19 commits it was showing were my fault: after you opened it, I rewrote the branch so all commits use the same identity. Some were authored as `abel`, which is the same person as `hotcache` but shows up as a different name in the commits. That would have looked like a third contributor here.
   
   The trees were byte-identical, but the rewrite changed all the SHAs from the merge commit onward, including `e7db3c8`. Your two commits rebased cleanly onto the new tip, so there's nothing you need to redo.
   
   ### Removed files are now `DELETED` entries — `8a4f780`
   
   Took this as described. `filter_manifests` now writes each removed entry with `add_delete_entry`, which stamps the removing snapshot while keeping the file's own sequence numbers.
   
   When a commit empties a manifest, we keep the manifest with just those deleted entries rather than dropping it immediately. The next merging commit removes it. This is the same rule `82ad3ae` already used for manifests that became all-deleted in an earlier snapshot.
   
   A few assertions needed to move with that behavior, so you don't need to change the fixture:
   
   * `test_rewrite_files_removes_empty_manifest` and `retires_the_old_spec` no longer expect the manifest to be absent. They check that it has no live entries and that a **later rewrite** drops it. My first attempt used an append and failed because the all-deleted filtering happens in `MergingSnapshotProducer`, not `FastAppend`. So an append keeps the manifest around, just like Java does. Only a merging commit rebuilds the manifest list. That's #2545 again.
   * `keeps_survivors_on_their_partition_spec` now expects the spec-0 manifest's partition summary to include the removed file as well as the survivor (`lower_bound` is the morning value rather than the evening one). This matches Java's `ManifestWriter#addEntry`, which updates the summary for every entry regardless of status.
   
   Reverting the change makes both of the first two tests fail, so they should help lock in the behavior.
   
   `transaction::rewrite` is now at 25 tests. `cargo test -p iceberg`, `cargo fmt`, `clippy -D warnings`, `cargo doc -D warnings`, and `make check-public-api` are all clean.
   
   ### Still open
   
   The interim conflict check is unchanged and still table-wide. The file-level `validateNoNewDeletesForDataFiles` is still PR6, on #2243.
   
   Partition-summary pruning, `set_commit_uuid` / snapshot properties, and the `SnapshotSummaryCollector::merge` default are all listed as follow-ups in the description.
   
   Thanks again for running this against Polaris with an independent reader. The missing `DELETED` entries weren't something the unit tests could catch — the summary reported `deleted-data-files=2`, and nothing looked wrong until Trino reported `deleted_data_files_count = 0`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] feat(transaction): add MergingSnapshotProducer and RewriteFilesAction [iceberg-rust]

Posted by "jopdorp (via GitHub)" <gi...@apache.org>.
jopdorp commented on PR #3046:
URL: https://github.com/apache/iceberg-rust/pull/3046#issuecomment-6417415334

   @laskoviymishka @JanKaul hotcache/iceberg-rust#3 has the remove-only rewrite you asked for, CI green, but hotcache has been away since September and this branch now conflicts with main. Want us to open a follow-up PR with hotcache's commits plus the rebase and that change, so you can review it there?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org