You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "qzyu999 (via GitHub)" <gi...@apache.org> on 2026/08/18 16:43:11 UTC
[PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
qzyu999 opened a new pull request, #3131:
URL: https://github.com/apache/iceberg-python/pull/3131
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
Closes #3130
# Rationale for this change
In a current PR (#3124, part of #1092), the proposed `replace()` API accepts a PyArrow dataframe (`pa.Table`), forcing the table engine to physically serialize data during a metadata transaction commit. This couples execution with the catalog, diverges from Java Iceberg's native `RewriteFiles` builder behavior, and fails to register under `Operation.REPLACE`.
This PR redesigns `table.replace()` and `transaction.replace()` to accept `Iterable[DataFile]` inputs. By externalizing physical data writing (e.g., compaction via Ray), the new explicit metadata-only `_RewriteFiles` SnapshotProducer can natively swap snapshot pointers in the manifests, perfectly inheriting ancestral sequence numbers for `DELETED` entries to ensure time-travel equivalence.
## Are these changes tested?
**Yes.**
Fully exhaustive test coverage has been added to `tests/table/test_replace.py`. The suite validates:
1. Context manager executions tracking valid history growth (`len(table.history())`).
2. Snapshot summary bindings asserting strict `Operation.REPLACE` tags.
3. Accurate evaluation of delta-metrics (added/deleted files and records tracking perfectly).
4. **Low-level serialization:** Bypassed high-level discard filters on `manifest.fetch_manifest_entry(discard_deleted=False)` to natively assert that `status=DELETED` overrides are accurately preserving avro sequence numbers.
5. Idempotent edge cases where `replace([], [])` successfully short-circuits the commit loop without mutating history.
## Are there any user-facing changes?
**Yes.**
The method signature for `Table.replace()` and `Transaction.replace()` has been updated from the original PR #3124.
It no longer accepts a PyArrow DataFrame (`df: pa.Table`). Instead, it now requests two arguments:
`files_to_delete: Iterable[DataFile]` and `files_to_add: Iterable[DataFile]`, following the convention seen in the Java implementation.
<!-- In the case of user-facing changes, please add the changelog label. -->
*(Please add the `changelog` label)*
## AI Disclosure
AI was used to help understand the code base and draft code changes. All code changes have been thoroughly reviewed, ensuring that the code changes are in line with a broader understanding of the codebase.
- Worth deeper review after AI-assistance:
- The `test_invalid_operation()` in `tests/table/test_snapshots.py` previously used `Operation.REPLACE` as a value to test invalid operations, but with this change `Operation.REPLACE` becomes valid. In place I just put a dummy Operation.
- The `_RewriteFiles` in `pyiceberg/table/update/snapshot.py` overrides the `_deleted_entries` and `_existing_manifests` functions. I sought to test this thoroughly that it was done correctly. I am thinking it's possible to improve the test suite to make this more rigorous. I am open to suggestions on how that could be done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3096185437
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -724,6 +790,14 @@ def delete(self) -> _DeleteFiles:
snapshot_properties=self._snapshot_properties,
)
+ def replace(self) -> _RewriteFiles:
+ return _RewriteFiles(
+ operation=Operation.REPLACE,
+ transaction=self._transaction,
Review Comment:
Hi @geruh, thanks for noticing, this was an oversight. The changes have been made in c3570d82c8cc974e5928fb9d1efa1e1a4bc0cdb1 and a test has been added for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-4263533082
Hi @geruh, thanks again for the helpful feedback, I've responded to each of your review comments and updated the code accordingly, with the exception on the note about naming convention for rewrite/replace as that is pending a response from @kevinjqliu, PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-4257371529
Hi @geruh, thanks for the awesome feedback, I've responded to each of your replies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3095873141
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +667,96 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValueError("Cannot delete files that are not present in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
Review Comment:
Okay let's keep the check I took a deeper look into the snapshot producer on the java side so let's align closer to that:
https://github.com/apache/iceberg/blob/dde712ec9ed6c9d28183ee4615d50f97b246af5d/core/src/main/java/org/apache/iceberg/SnapshotProducer.java#L322-L334
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3089060985
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -724,6 +814,14 @@ def delete(self) -> _DeleteFiles:
snapshot_properties=self._snapshot_properties,
)
+ def replace(self) -> _RewriteFiles:
+ return _RewriteFiles(
Review Comment:
I'm sort of confused by the naming since we are introducing a user facing API `replace` but the underlying snapshot operation is a `rewrite`? We should rename to `rewrite()` for consistency? Unless I'm missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3090123085
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +667,96 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
Review Comment:
Hi @geruh, great suggestion, I've applied the changes in c8162a8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3002985921
##########
pyiceberg/table/__init__.py:
##########
@@ -450,6 +450,32 @@ def update_statistics(self) -> UpdateStatistics:
"""
return UpdateStatistics(transaction=self)
+ def replace(
Review Comment:
Hi @kevinjqliu, I think that logic makes sense, as it would be dangerous for users to use these without being able to enforce the underlying expectations of the values input to these functions. Removed them in 94bd87e1d64f8ee524dd46b789e90eacbe471076
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3090758831
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +667,96 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValueError("Cannot delete files that are not present in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
Review Comment:
Hi @geruh, thanks for flagging this, you're right that this is a safety guard, but it doesn't yet factor in future changes when adding delete file rewriting. Should we add something like this?
```python
# Note: This physical record count invariant is a sanity guard for data file
# compaction to ensure no data is accidentally duplicated or invented.
# TODO: This will need to be evolved into a logical record count validation
# once PyIceberg supports rewriting delete files (Merge-on-Read).
added_records = sum(f.record_count for f in self._added_data_files)
deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
if added_records > deleted_records:
raise ValueError(f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})")
```
This logical record count validation would involve something like having the `_commit` method to do the following, which the codebase currently cannot do:
- Identify associated Delete Files: For every `DataFile` you are deleting, you would need to find every Position Delete or Equality Delete file that points to it.
- Calculate the "Subtraction": You would need to subtract those delete row counts from the physical record_count of the old files to find the Old Logical Count.
- Compare: You would then verify that Old Logical Count == New Logical Count.
The current `_RewriteFiles` implementation is "blind" to deletes. It only tracks `_added_data_files` and `_deleted_data_files`.
I believe this can be part of a full MoR implementation, something that I would love to work on after finishing these maintenance tasks.
Otherwise, I can also remove it from `_RerwriteFiles` and move forward, WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3096190054
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +669,70 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
Review Comment:
Hi @geruh, the changes have been made in c3570d82c8cc974e5928fb9d1efa1e1a4bc0cdb1 as part of adding `branch` as an arg for `_RewriteFiles`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-5075870004
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-4714093059
Still waiting on another PR in review. This may also require some DataFusion integration to handle OOM issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-5008820820
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3096281892
##########
tests/table/test_replace.py:
##########
@@ -0,0 +1,637 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import cast
+
+import pytest
+
+from pyiceberg.catalog import Catalog
+from pyiceberg.manifest import (
+ DataFile,
+ DataFileContent,
+ FileFormat,
+ ManifestEntry,
+ ManifestEntryStatus,
+)
+from pyiceberg.schema import Schema
+from pyiceberg.table.snapshots import Operation, Snapshot, Summary
+from pyiceberg.typedef import Record
+
+
+def test_replace_internally(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace",
+ schema=Schema(),
+ )
+
+ # 1. File we will delete
+ file_to_delete = DataFile.from_args(
Review Comment:
Hi @geruh, great idea, this has been implemented in 9681ec3369e7b605d98d3a87027a0c434ef0ccc2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3937594609
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -852,6 +850,120 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files.
+
+ Produces a REPLACE snapshot that swaps existing data files for new ones without
+ changing the logical contents of the table. This is the metadata-only operation
+ used by compaction (bin-packing, sort, format migration).
+
+ Current scope:
+ - Data file rewriting only (delete + add DataFiles)
+ - Validates: files-to-delete exist, added_records <= deleted_records,
+ no new delete files conflict with replaced data files
+
+ Future work (additive — no structural changes needed):
+ - Delete-file rewriting (add _deleted_delete_files set + separate manifest handling)
+ - dataSequenceNumber override (pin new files' seq to match replaced, for eq-delete safety)
+ - validateFromSnapshot (expose _starting_snapshot_id setter for long-running planners)
+ - ignoreEqualityDeletes in validation (coupled with dataSequenceNumber)
+ """
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
Review Comment:
```suggestion
if not self._deleted_data_files and self._added_data_files:
return (), ()
```
With this, you can avoid all of the nesting. In general, I think this method could be cleaned up into a series of `if invariant, throw error` statements, rather than nesting.
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -852,6 +850,120 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files.
+
+ Produces a REPLACE snapshot that swaps existing data files for new ones without
+ changing the logical contents of the table. This is the metadata-only operation
+ used by compaction (bin-packing, sort, format migration).
+
+ Current scope:
+ - Data file rewriting only (delete + add DataFiles)
+ - Validates: files-to-delete exist, added_records <= deleted_records,
+ no new delete files conflict with replaced data files
+
+ Future work (additive — no structural changes needed):
+ - Delete-file rewriting (add _deleted_delete_files set + separate manifest handling)
+ - dataSequenceNumber override (pin new files' seq to match replaced, for eq-delete safety)
+ - validateFromSnapshot (expose _starting_snapshot_id setter for long-running planners)
+ - ignoreEqualityDeletes in validation (coupled with dataSequenceNumber)
+ """
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValidationException("Cannot commit, missing data files to be rewritten that are not in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
+ raise ValidationException(
+ f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})"
+ )
+
+ return super()._commit()
+ else:
+ return (), ()
+
+ @cached_property
+ def _cached_deleted_entries(self) -> list[ManifestEntry]:
Review Comment:
This is near-identical to _OverwriteFiles._deleted_entries.
What if we brought this up to the base class and then just had each subclass have their own version of _get_entries?
That way, we can have the base class handle the caching portion too.
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -852,6 +850,120 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files.
+
+ Produces a REPLACE snapshot that swaps existing data files for new ones without
+ changing the logical contents of the table. This is the metadata-only operation
+ used by compaction (bin-packing, sort, format migration).
+
+ Current scope:
+ - Data file rewriting only (delete + add DataFiles)
+ - Validates: files-to-delete exist, added_records <= deleted_records,
+ no new delete files conflict with replaced data files
+
+ Future work (additive — no structural changes needed):
+ - Delete-file rewriting (add _deleted_delete_files set + separate manifest handling)
+ - dataSequenceNumber override (pin new files' seq to match replaced, for eq-delete safety)
+ - validateFromSnapshot (expose _starting_snapshot_id setter for long-running planners)
+ - ignoreEqualityDeletes in validation (coupled with dataSequenceNumber)
+ """
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValidationException("Cannot commit, missing data files to be rewritten that are not in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
+ raise ValidationException(
+ f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})"
+ )
+
+ return super()._commit()
+ else:
+ return (), ()
+
+ @cached_property
+ def _cached_deleted_entries(self) -> list[ManifestEntry]:
+ """Build manifest entries marking deleted data files with DELETED status."""
+ if self._parent_snapshot_id is not None:
+ previous_snapshot = self._transaction.table_metadata.snapshot_by_id(self._parent_snapshot_id)
+ if previous_snapshot is None:
+ raise ValueError(f"Could not find the previous snapshot: {self._parent_snapshot_id}")
+
+ executor = ExecutorFactory.get_or_create()
+
+ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
+ return [
+ ManifestEntry.from_args(
+ status=ManifestEntryStatus.DELETED,
+ snapshot_id=self.snapshot_id,
+ sequence_number=entry.sequence_number,
+ file_sequence_number=entry.file_sequence_number,
+ data_file=entry.data_file,
+ )
+ for entry in manifest.fetch_manifest_entry(self._io, discard_deleted=True)
+ if entry.data_file.content == DataFileContent.DATA and entry.data_file in self._deleted_data_files
+ ]
+
+ list_of_entries = executor.map(_get_entries, previous_snapshot.manifests(self._io))
+ return list(itertools.chain(*list_of_entries))
+ else:
+ return []
+
+ def _deleted_entries(self) -> list[ManifestEntry]:
+ return self._cached_deleted_entries
+
+ def _existing_manifests(self) -> list[ManifestFile]:
+ return self._get_existing_manifests()
Review Comment:
Can you manually pass in `should_use_manifest_pruning` and then remove the default? The default is only used once and it makes it harder to read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3096424562
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -165,6 +165,50 @@ def _calculate_added_rows(self, manifests: list[ManifestFile]) -> int:
added_rows += manifest.added_rows_count
return added_rows
+ def _get_existing_manifests(self) -> list[ManifestFile]:
+ """Filter existing manifests and rewrite those containing deleted data files."""
+ existing_files: list[ManifestFile] = []
+ # Use manifest pruning if a predicate is set (primarily for Overwrite)
+ manifest_evaluators: dict[int, Callable[[ManifestFile], bool]] = KeyDefaultDict(self._build_manifest_evaluator)
+
+ if snapshot := self._transaction.table_metadata.snapshot_by_name(name=self._target_branch):
+ for manifest_file in snapshot.manifests(io=self._io):
+ # Skip pruning for rewrite operations unless we want to optimize later
+ if self._operation == Operation.OVERWRITE and not manifest_evaluators[manifest_file.partition_spec_id](
Review Comment:
Hi @geruh, great catch, I agree that the base class shouldn't be coupled to specific child operations. I've refactored `_get_existing_manifests` to accept a `should_use_manifest_pruning` flag so the child classes can explicitly opt-in. The updates are in 8f1f9b949ebcc83e99ccc80e0b3d9fda050a096d.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3095389329
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -724,6 +790,14 @@ def delete(self) -> _DeleteFiles:
snapshot_properties=self._snapshot_properties,
)
+ def replace(self) -> _RewriteFiles:
+ return _RewriteFiles(
+ operation=Operation.REPLACE,
+ transaction=self._transaction,
Review Comment:
I noticed that branch is missing here is there a reason for that?
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -724,6 +814,14 @@ def delete(self) -> _DeleteFiles:
snapshot_properties=self._snapshot_properties,
)
+ def replace(self) -> _RewriteFiles:
+ return _RewriteFiles(
Review Comment:
Yeah there is a bit of a distinction here, since rewrite is basically the rewrite of data files and replace is the logical change to your snapshot metadata. My thinking is that the users in java today are used to interacting with this api through:
```java
table.newRewrite()
.deleteFile(old)
.addFile(new)
.commit();
```
So someone coming from Java Iceberg will look for rewrite, not replace. But ultimately maybe there is more of a history as to why the it follows this naming convention im missing on.
WDYT @kevinjqliu?
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +669,70 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValueError("Cannot delete files that are not present in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
+ raise ValueError(f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})")
+
+ return super()._commit()
+ else:
+ return (), ()
+
+ @cached_property
+ def _cached_deleted_entries(self) -> list[ManifestEntry]:
+ """Check if we need to mark the files as deleted."""
+ if self._parent_snapshot_id is not None:
+ previous_snapshot = self._transaction.table_metadata.snapshot_by_id(self._parent_snapshot_id)
+ if previous_snapshot is None:
+ raise ValueError(f"Could not find the previous snapshot: {self._parent_snapshot_id}")
+
+ executor = ExecutorFactory.get_or_create()
+
+ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
+ return [
+ ManifestEntry.from_args(
+ status=ManifestEntryStatus.DELETED,
+ snapshot_id=self.snapshot_id,
+ sequence_number=entry.sequence_number,
+ file_sequence_number=entry.file_sequence_number,
+ data_file=entry.data_file,
+ )
+ for entry in manifest.fetch_manifest_entry(self._io, discard_deleted=True)
+ if entry.data_file.content == DataFileContent.DATA and entry.data_file in self._deleted_data_files
+ ]
+
+ list_of_entries = executor.map(_get_entries, previous_snapshot.manifests(self._io))
+ return list(itertools.chain(*list_of_entries))
+ else:
+ return []
+
+ def _deleted_entries(self) -> list[ManifestEntry]:
+ """Check if we need to mark the files as deleted."""
+ return self._cached_deleted_entries
+
+ def _existing_manifests(self) -> list[ManifestFile]:
+ """To determine if there are any existing manifests."""
+ return self._get_existing_manifests()
Review Comment:
nit: Looks like these doc strings were copy pasta'd over from the other classes, and don't fit how they are used here. Either we can remove them or change to fit their usage.
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -165,6 +165,50 @@ def _calculate_added_rows(self, manifests: list[ManifestFile]) -> int:
added_rows += manifest.added_rows_count
return added_rows
+ def _get_existing_manifests(self) -> list[ManifestFile]:
+ """Filter existing manifests and rewrite those containing deleted data files."""
+ existing_files: list[ManifestFile] = []
+ # Use manifest pruning if a predicate is set (primarily for Overwrite)
+ manifest_evaluators: dict[int, Callable[[ManifestFile], bool]] = KeyDefaultDict(self._build_manifest_evaluator)
+
+ if snapshot := self._transaction.table_metadata.snapshot_by_name(name=self._target_branch):
+ for manifest_file in snapshot.manifests(io=self._io):
+ # Skip pruning for rewrite operations unless we want to optimize later
+ if self._operation == Operation.OVERWRITE and not manifest_evaluators[manifest_file.partition_spec_id](
Review Comment:
I'm not too keen on the child classes information leaking into the base here. How about we use an argument like `should_use_manifest_pruning` or something along those lines.
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +669,70 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
Review Comment:
nit: we can remove the constructor here
##########
tests/table/test_replace.py:
##########
@@ -0,0 +1,637 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import cast
+
+import pytest
+
+from pyiceberg.catalog import Catalog
+from pyiceberg.manifest import (
+ DataFile,
+ DataFileContent,
+ FileFormat,
+ ManifestEntry,
+ ManifestEntryStatus,
+)
+from pyiceberg.schema import Schema
+from pyiceberg.table.snapshots import Operation, Snapshot, Summary
+from pyiceberg.typedef import Record
+
+
+def test_replace_internally(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace",
+ schema=Schema(),
+ )
+
+ # 1. File we will delete
+ file_to_delete = DataFile.from_args(
Review Comment:
nit: extract this into a helper or see if one exists already.
##########
tests/table/test_snapshots.py:
##########
@@ -398,8 +398,8 @@ def test_merge_snapshot_summaries_overwrite_summary() -> None:
def test_invalid_operation() -> None:
with pytest.raises(ValueError) as e:
- update_snapshot_summaries(summary=Summary(Operation.REPLACE))
- assert "Operation not implemented: Operation.REPLACE" in str(e.value)
+ update_snapshot_summaries(summary=Summary.model_construct(operation="unknown_operation"))
Review Comment:
Makes sense if we have support :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3090684619
##########
tests/table/test_replace.py:
##########
@@ -0,0 +1,458 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import cast
+
+import pytest
+
+from pyiceberg.catalog import Catalog
+from pyiceberg.manifest import (
+ DataFile,
+ DataFileContent,
+ FileFormat,
+ ManifestEntry,
+ ManifestEntryStatus,
+)
+from pyiceberg.schema import Schema
+from pyiceberg.table.snapshots import Operation, Snapshot, Summary
+from pyiceberg.typedef import Record
+
+
+def test_replace_internally(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace",
+ schema=Schema(),
+ )
+
+ # 1. File we will delete
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ # 2. File we will leave completely untouched
+ file_to_keep = DataFile.from_args(
+ file_path="s3://bucket/test/data/kept.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=50,
+ file_size_in_bytes=512,
+ content=DataFileContent.DATA,
+ )
+ file_to_keep.spec_id = 0
+
+ # 3. File we are adding as a replacement
+ file_to_add = DataFile.from_args(
+ file_path="s3://bucket/test/data/added.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_add.spec_id = 0
+
+ # Initially append BOTH the file to delete and the file to keep
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+ append_snapshot.append_data_file(file_to_keep)
+
+ old_snapshot = cast(Snapshot, table.current_snapshot())
+ old_snapshot_id = old_snapshot.snapshot_id
+ old_sequence_number = cast(int, old_snapshot.sequence_number)
+
+ # Call the internal replace API
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_to_delete)
+ rewrite.append_data_file(file_to_add)
+
+ snapshot = cast(Snapshot, table.current_snapshot())
+ summary = cast(Summary, snapshot.summary)
+
+ # 1. Has a unique snapshot ID
+ assert snapshot.snapshot_id is not None
+ assert snapshot.snapshot_id != old_snapshot_id
+
+ # 2. Parent points to the previous snapshot
+ assert snapshot.parent_snapshot_id == old_snapshot_id
+
+ # 3. Sequence number is exactly previous + 1
+ assert snapshot.sequence_number == old_sequence_number + 1
+
+ # 4. Operation type is set to "replace"
+ assert summary["operation"] == Operation.REPLACE
+
+ # 5. Manifest list path is correct (just verify it exists and is a string path)
+ assert snapshot.manifest_list is not None
+ assert isinstance(snapshot.manifest_list, str)
+
+ # 6. Summary counts are accurate
+ assert summary["added-data-files"] == "1"
+ assert summary["deleted-data-files"] == "1"
+ assert summary["added-records"] == "100"
+ assert summary["deleted-records"] == "100"
+ assert summary["total-records"] == "150"
+
+ # Fetch all entries from the new manifests
+ manifest_files = snapshot.manifests(table.io)
+ entries: list[ManifestEntry] = []
+ for manifest in manifest_files:
+ entries.extend(manifest.fetch_manifest_entry(table.io, discard_deleted=False))
+
+ # We expect 3 entries: ADDED, DELETED, and EXISTING
+ assert len(entries) == 3
+
+ # Check ADDED
+ added_entries = [e for e in entries if e.status == ManifestEntryStatus.ADDED]
+ assert len(added_entries) == 1
+ assert added_entries[0].data_file.file_path == file_to_add.file_path
+ assert added_entries[0].snapshot_id == snapshot.snapshot_id
+
+ # Check DELETED
+ deleted_entries = [e for e in entries if e.status == ManifestEntryStatus.DELETED]
+ assert len(deleted_entries) == 1
+ assert deleted_entries[0].data_file.file_path == file_to_delete.file_path
+ assert deleted_entries[0].snapshot_id == snapshot.snapshot_id
+
+ # Check EXISTING
+ existing_entries = [e for e in entries if e.status == ManifestEntryStatus.EXISTING]
+ assert len(existing_entries) == 1
+ assert existing_entries[0].data_file.file_path == file_to_keep.file_path
+ assert existing_entries[0].snapshot_id == old_snapshot_id
+
+
+def test_replace_reuses_unaffected_manifests(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_reuse_manifest",
+ schema=Schema(),
+ )
+
+ file_a = DataFile.from_args(
+ file_path="s3://bucket/test/data/a.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_a.spec_id = 0
+
+ file_b = DataFile.from_args(
+ file_path="s3://bucket/test/data/b.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_b.spec_id = 0
+
+ file_c = DataFile.from_args(
+ file_path="s3://bucket/test/data/c.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_c.spec_id = 0
+
+ # Commit 1: Append file A (Creates Manifest 1)
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_a)
+
+ # Commit 2: Append file B (Creates Manifest 2)
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_b)
+
+ snapshot_before = cast(Snapshot, table.current_snapshot())
+ manifests_before = snapshot_before.manifests(table.io)
+ assert len(manifests_before) == 2
+
+ # Identify which manifest belongs to file_b and file_a
+ manifest_b_path = None
+ manifest_a_path = None
+ for m in manifests_before:
+ entries = m.fetch_manifest_entry(table.io, discard_deleted=False)
+ if any(e.data_file.file_path == file_b.file_path for e in entries):
+ manifest_b_path = m.manifest_path
+ if any(e.data_file.file_path == file_a.file_path for e in entries):
+ manifest_a_path = m.manifest_path
+
+ assert manifest_b_path is not None
+ assert manifest_a_path is not None
+
+ # Commit 3: Replace file A with file C
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_a)
+ rewrite.append_data_file(file_c)
+
+ snapshot_after = cast(Snapshot, table.current_snapshot())
+ assert snapshot_after is not None
+ manifests_after = snapshot_after.manifests(table.io)
+
+ # We expect 3 manifests:
+ # 1. The reused one for file B
+ # 2. The newly rewritten one marking file A as DELETED
+ # 3. The new one for file C (ADDED)
+ assert len(manifests_after) == 3
+
+ manifest_paths_after = [m.manifest_path for m in manifests_after]
+
+ # ASSERTION 1: The untouched manifest is completely reused (the path matches exactly)
+ assert manifest_b_path in manifest_paths_after
+
+ # ASSERTION 2: File A's old manifest is NOT reused (since it was rewritten to change status to DELETED)
+ assert manifest_a_path not in manifest_paths_after
+
+
+def test_replace_empty_files(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_empty",
+ schema=Schema(),
+ )
+
+ # Replacing empty lists should not throw errors, but should produce no changes.
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace():
+ pass # Entering and exiting the context manager without adding/deleting
+
+ # History should be completely empty since no files were rewritten
+ assert len(table.history()) == 0
+ assert table.current_snapshot() is None
+
+
+def test_replace_missing_file_abort(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_missing",
+ schema=Schema(),
+ )
+
+ fake_data_file = DataFile.from_args(
+ file_path="s3://bucket/test/data/does_not_exist.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ fake_data_file.spec_id = 0
+
+ new_data_file = DataFile.from_args(
+ file_path="s3://bucket/test/data/new.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ new_data_file.spec_id = 0
+
+ # Ensure it aborts when trying to replace a file that isn't in the table
+ with pytest.raises(ValueError, match="Cannot delete files that are not present in the table"):
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(fake_data_file)
+ rewrite.append_data_file(new_data_file)
+
+
+def test_replace_invariant_violation(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_invariant",
+ schema=Schema(),
+ )
+
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ # Create a new file with MORE records than the one we are deleting
+ too_many_records_file = DataFile.from_args(
+ file_path="s3://bucket/test/data/too_many.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=101,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ too_many_records_file.spec_id = 0
+
+ # Initially append to have something to replace
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+
+ # Ensure it enforces the invariant: records added <= records removed
+ with pytest.raises(ValueError, match=r"Invalid replace: records added \(101\) exceeds records removed \(100\)"):
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_to_delete)
+ rewrite.append_data_file(too_many_records_file)
+
+
+def test_replace_allows_shrinking_for_soft_deletes(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_shrink",
+ schema=Schema(),
+ )
+
+ # Old data file has 100 records
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ # New data file only has 90 records (simulating 10 records were soft-deleted)
+ shrunk_file_to_add = DataFile.from_args(
+ file_path="s3://bucket/test/data/shrunk.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=90,
+ file_size_in_bytes=900,
+ content=DataFileContent.DATA,
+ )
+ shrunk_file_to_add.spec_id = 0
+
+ # Initially append
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+
+ # This should succeed without throwing an invariant violation
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_to_delete)
+ rewrite.append_data_file(shrunk_file_to_add)
+
+ snapshot = cast(Snapshot, table.current_snapshot())
+ summary = cast(Summary, snapshot.summary)
+
+ assert summary["operation"] == Operation.REPLACE
+ assert summary["added-records"] == "90"
+ assert summary["deleted-records"] == "100"
+
+
+def test_replace_passes_through_delete_manifests(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_delete_manifests",
+ schema=Schema(),
+ properties={"format-version": "2"},
+ )
+
+ # 1. Data file we will replace
+ file_a = DataFile.from_args(
+ file_path="s3://bucket/test/data/a.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_a.spec_id = 0
+
+ # 2. A Position Delete file (representing row-level deletes)
+ file_a_deletes = DataFile.from_args(
+ file_path="s3://bucket/test/data/a_deletes.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=2,
+ file_size_in_bytes=50,
+ content=DataFileContent.POSITION_DELETES,
+ )
+ file_a_deletes.spec_id = 0
+
+ # 3. Data file we are adding as a replacement
+ file_b = DataFile.from_args(
+ file_path="s3://bucket/test/data/b.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_b.spec_id = 0
+
+ # Commit 1: Append the data file
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_a)
+
+ # Commit 2: Append the delete file
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_a_deletes)
+
+ # Find the path of the delete manifest so we can verify it survives
+ snapshot_before = cast(Snapshot, table.current_snapshot())
+ manifests_before = snapshot_before.manifests(table.io)
+
+ delete_manifest_path = None
+ for m in manifests_before:
+ entries = m.fetch_manifest_entry(table.io, discard_deleted=False)
+ if any(e.data_file.file_path == file_a_deletes.file_path for e in entries):
+ delete_manifest_path = m.manifest_path
+ break
+
+ assert delete_manifest_path is not None
+
+ # Commit 3: Replace data file A with data file B
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_a)
+ rewrite.append_data_file(file_b)
+
+ # Verify the delete manifest was passed through unchanged
+ snapshot_after = cast(Snapshot, table.current_snapshot())
+ assert snapshot_after is not None
+ manifests_after = snapshot_after.manifests(table.io)
+ manifest_paths_after = [m.manifest_path for m in manifests_after]
+
+ assert delete_manifest_path in manifest_paths_after
Review Comment:
Hi @geruh, I've added a test to check the sequence numbers in d939b67.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-6103819281
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3090684619
##########
tests/table/test_replace.py:
##########
@@ -0,0 +1,458 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import cast
+
+import pytest
+
+from pyiceberg.catalog import Catalog
+from pyiceberg.manifest import (
+ DataFile,
+ DataFileContent,
+ FileFormat,
+ ManifestEntry,
+ ManifestEntryStatus,
+)
+from pyiceberg.schema import Schema
+from pyiceberg.table.snapshots import Operation, Snapshot, Summary
+from pyiceberg.typedef import Record
+
+
+def test_replace_internally(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace",
+ schema=Schema(),
+ )
+
+ # 1. File we will delete
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ # 2. File we will leave completely untouched
+ file_to_keep = DataFile.from_args(
+ file_path="s3://bucket/test/data/kept.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=50,
+ file_size_in_bytes=512,
+ content=DataFileContent.DATA,
+ )
+ file_to_keep.spec_id = 0
+
+ # 3. File we are adding as a replacement
+ file_to_add = DataFile.from_args(
+ file_path="s3://bucket/test/data/added.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_add.spec_id = 0
+
+ # Initially append BOTH the file to delete and the file to keep
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+ append_snapshot.append_data_file(file_to_keep)
+
+ old_snapshot = cast(Snapshot, table.current_snapshot())
+ old_snapshot_id = old_snapshot.snapshot_id
+ old_sequence_number = cast(int, old_snapshot.sequence_number)
+
+ # Call the internal replace API
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_to_delete)
+ rewrite.append_data_file(file_to_add)
+
+ snapshot = cast(Snapshot, table.current_snapshot())
+ summary = cast(Summary, snapshot.summary)
+
+ # 1. Has a unique snapshot ID
+ assert snapshot.snapshot_id is not None
+ assert snapshot.snapshot_id != old_snapshot_id
+
+ # 2. Parent points to the previous snapshot
+ assert snapshot.parent_snapshot_id == old_snapshot_id
+
+ # 3. Sequence number is exactly previous + 1
+ assert snapshot.sequence_number == old_sequence_number + 1
+
+ # 4. Operation type is set to "replace"
+ assert summary["operation"] == Operation.REPLACE
+
+ # 5. Manifest list path is correct (just verify it exists and is a string path)
+ assert snapshot.manifest_list is not None
+ assert isinstance(snapshot.manifest_list, str)
+
+ # 6. Summary counts are accurate
+ assert summary["added-data-files"] == "1"
+ assert summary["deleted-data-files"] == "1"
+ assert summary["added-records"] == "100"
+ assert summary["deleted-records"] == "100"
+ assert summary["total-records"] == "150"
+
+ # Fetch all entries from the new manifests
+ manifest_files = snapshot.manifests(table.io)
+ entries: list[ManifestEntry] = []
+ for manifest in manifest_files:
+ entries.extend(manifest.fetch_manifest_entry(table.io, discard_deleted=False))
+
+ # We expect 3 entries: ADDED, DELETED, and EXISTING
+ assert len(entries) == 3
+
+ # Check ADDED
+ added_entries = [e for e in entries if e.status == ManifestEntryStatus.ADDED]
+ assert len(added_entries) == 1
+ assert added_entries[0].data_file.file_path == file_to_add.file_path
+ assert added_entries[0].snapshot_id == snapshot.snapshot_id
+
+ # Check DELETED
+ deleted_entries = [e for e in entries if e.status == ManifestEntryStatus.DELETED]
+ assert len(deleted_entries) == 1
+ assert deleted_entries[0].data_file.file_path == file_to_delete.file_path
+ assert deleted_entries[0].snapshot_id == snapshot.snapshot_id
+
+ # Check EXISTING
+ existing_entries = [e for e in entries if e.status == ManifestEntryStatus.EXISTING]
+ assert len(existing_entries) == 1
+ assert existing_entries[0].data_file.file_path == file_to_keep.file_path
+ assert existing_entries[0].snapshot_id == old_snapshot_id
+
+
+def test_replace_reuses_unaffected_manifests(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_reuse_manifest",
+ schema=Schema(),
+ )
+
+ file_a = DataFile.from_args(
+ file_path="s3://bucket/test/data/a.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_a.spec_id = 0
+
+ file_b = DataFile.from_args(
+ file_path="s3://bucket/test/data/b.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_b.spec_id = 0
+
+ file_c = DataFile.from_args(
+ file_path="s3://bucket/test/data/c.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_c.spec_id = 0
+
+ # Commit 1: Append file A (Creates Manifest 1)
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_a)
+
+ # Commit 2: Append file B (Creates Manifest 2)
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_b)
+
+ snapshot_before = cast(Snapshot, table.current_snapshot())
+ manifests_before = snapshot_before.manifests(table.io)
+ assert len(manifests_before) == 2
+
+ # Identify which manifest belongs to file_b and file_a
+ manifest_b_path = None
+ manifest_a_path = None
+ for m in manifests_before:
+ entries = m.fetch_manifest_entry(table.io, discard_deleted=False)
+ if any(e.data_file.file_path == file_b.file_path for e in entries):
+ manifest_b_path = m.manifest_path
+ if any(e.data_file.file_path == file_a.file_path for e in entries):
+ manifest_a_path = m.manifest_path
+
+ assert manifest_b_path is not None
+ assert manifest_a_path is not None
+
+ # Commit 3: Replace file A with file C
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_a)
+ rewrite.append_data_file(file_c)
+
+ snapshot_after = cast(Snapshot, table.current_snapshot())
+ assert snapshot_after is not None
+ manifests_after = snapshot_after.manifests(table.io)
+
+ # We expect 3 manifests:
+ # 1. The reused one for file B
+ # 2. The newly rewritten one marking file A as DELETED
+ # 3. The new one for file C (ADDED)
+ assert len(manifests_after) == 3
+
+ manifest_paths_after = [m.manifest_path for m in manifests_after]
+
+ # ASSERTION 1: The untouched manifest is completely reused (the path matches exactly)
+ assert manifest_b_path in manifest_paths_after
+
+ # ASSERTION 2: File A's old manifest is NOT reused (since it was rewritten to change status to DELETED)
+ assert manifest_a_path not in manifest_paths_after
+
+
+def test_replace_empty_files(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_empty",
+ schema=Schema(),
+ )
+
+ # Replacing empty lists should not throw errors, but should produce no changes.
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace():
+ pass # Entering and exiting the context manager without adding/deleting
+
+ # History should be completely empty since no files were rewritten
+ assert len(table.history()) == 0
+ assert table.current_snapshot() is None
+
+
+def test_replace_missing_file_abort(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_missing",
+ schema=Schema(),
+ )
+
+ fake_data_file = DataFile.from_args(
+ file_path="s3://bucket/test/data/does_not_exist.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ fake_data_file.spec_id = 0
+
+ new_data_file = DataFile.from_args(
+ file_path="s3://bucket/test/data/new.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ new_data_file.spec_id = 0
+
+ # Ensure it aborts when trying to replace a file that isn't in the table
+ with pytest.raises(ValueError, match="Cannot delete files that are not present in the table"):
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(fake_data_file)
+ rewrite.append_data_file(new_data_file)
+
+
+def test_replace_invariant_violation(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_invariant",
+ schema=Schema(),
+ )
+
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ # Create a new file with MORE records than the one we are deleting
+ too_many_records_file = DataFile.from_args(
+ file_path="s3://bucket/test/data/too_many.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=101,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ too_many_records_file.spec_id = 0
+
+ # Initially append to have something to replace
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+
+ # Ensure it enforces the invariant: records added <= records removed
+ with pytest.raises(ValueError, match=r"Invalid replace: records added \(101\) exceeds records removed \(100\)"):
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_to_delete)
+ rewrite.append_data_file(too_many_records_file)
+
+
+def test_replace_allows_shrinking_for_soft_deletes(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_shrink",
+ schema=Schema(),
+ )
+
+ # Old data file has 100 records
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ # New data file only has 90 records (simulating 10 records were soft-deleted)
+ shrunk_file_to_add = DataFile.from_args(
+ file_path="s3://bucket/test/data/shrunk.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=90,
+ file_size_in_bytes=900,
+ content=DataFileContent.DATA,
+ )
+ shrunk_file_to_add.spec_id = 0
+
+ # Initially append
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+
+ # This should succeed without throwing an invariant violation
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_to_delete)
+ rewrite.append_data_file(shrunk_file_to_add)
+
+ snapshot = cast(Snapshot, table.current_snapshot())
+ summary = cast(Summary, snapshot.summary)
+
+ assert summary["operation"] == Operation.REPLACE
+ assert summary["added-records"] == "90"
+ assert summary["deleted-records"] == "100"
+
+
+def test_replace_passes_through_delete_manifests(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_delete_manifests",
+ schema=Schema(),
+ properties={"format-version": "2"},
+ )
+
+ # 1. Data file we will replace
+ file_a = DataFile.from_args(
+ file_path="s3://bucket/test/data/a.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_a.spec_id = 0
+
+ # 2. A Position Delete file (representing row-level deletes)
+ file_a_deletes = DataFile.from_args(
+ file_path="s3://bucket/test/data/a_deletes.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=2,
+ file_size_in_bytes=50,
+ content=DataFileContent.POSITION_DELETES,
+ )
+ file_a_deletes.spec_id = 0
+
+ # 3. Data file we are adding as a replacement
+ file_b = DataFile.from_args(
+ file_path="s3://bucket/test/data/b.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_b.spec_id = 0
+
+ # Commit 1: Append the data file
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_a)
+
+ # Commit 2: Append the delete file
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_a_deletes)
+
+ # Find the path of the delete manifest so we can verify it survives
+ snapshot_before = cast(Snapshot, table.current_snapshot())
+ manifests_before = snapshot_before.manifests(table.io)
+
+ delete_manifest_path = None
+ for m in manifests_before:
+ entries = m.fetch_manifest_entry(table.io, discard_deleted=False)
+ if any(e.data_file.file_path == file_a_deletes.file_path for e in entries):
+ delete_manifest_path = m.manifest_path
+ break
+
+ assert delete_manifest_path is not None
+
+ # Commit 3: Replace data file A with data file B
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_a)
+ rewrite.append_data_file(file_b)
+
+ # Verify the delete manifest was passed through unchanged
+ snapshot_after = cast(Snapshot, table.current_snapshot())
+ assert snapshot_after is not None
+ manifests_after = snapshot_after.manifests(table.io)
+ manifest_paths_after = [m.manifest_path for m in manifests_after]
+
+ assert delete_manifest_path in manifest_paths_after
Review Comment:
Hi @geruh, great idea, I've added a test to check the sequence numbers in d939b67.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3004042734
##########
tests/table/test_replace.py:
##########
@@ -0,0 +1,109 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pyiceberg.catalog import Catalog
+from pyiceberg.manifest import DataFile, DataFileContent, FileFormat
+from pyiceberg.schema import Schema
+from pyiceberg.table.snapshots import Operation
+from pyiceberg.typedef import Record
+
+
+def test_replace_api(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace",
+ schema=Schema(),
+ )
+
+ # Create mock DataFiles for the test
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ file_to_add = DataFile.from_args(
+ file_path="s3://bucket/test/data/added.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_add.spec_id = 0
+
+ # Initially append to have something to replace
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+
+ # Verify initial append snapshot
+ assert len(table.history()) == 1
+ snapshot = table.current_snapshot()
+ assert snapshot is not None
+ assert snapshot.summary is not None
+ assert snapshot.summary["operation"] == Operation.APPEND
+
+ # Call the replace API
+ table.replace(files_to_delete=[file_to_delete], files_to_add=[file_to_add])
+
+ # Verify the replacement created a REPLACE snapshot
+ assert len(table.history()) == 2
+ snapshot = table.current_snapshot()
+ assert snapshot is not None
+ assert snapshot.summary is not None
+ assert snapshot.summary["operation"] == Operation.REPLACE
+
+ # Verify the correct files are added and deleted
+ # The summary property tracks these counts
+ assert snapshot.summary["added-data-files"] == "1"
+ assert snapshot.summary["deleted-data-files"] == "1"
+ assert snapshot.summary["added-records"] == "100"
+ assert snapshot.summary["deleted-records"] == "100"
+
+ # Verify the new file exists in the new manifest
+ manifest_files = snapshot.manifests(table.io)
+ assert len(manifest_files) == 2 # One for ADDED, one for DELETED
+
+ # Check that sequence numbers were handled properly natively by verifying the manifest contents
+ entries = []
+ for manifest in manifest_files:
+ for entry in manifest.fetch_manifest_entry(table.io, discard_deleted=False):
+ entries.append(entry)
+
+ # One entry for ADDED (new file), one for DELETED (old file)
+ assert len(entries) == 2
Review Comment:
Hi @kevinjqliu, I've addressed this in the latest 33aaef0817b1366d80bcd8194a0c2ca5ba6f46f2, where the status of each file is tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-4468114397
As mentioned in #3319, I would like to delay the merging of this PR #3131 until the completion of #3320. The reason is that there is a `_validate_concurrency()` function that is being developed there, and upon completion it would make sense to integrate it `_RewriteFiles` to run all conflict validations before trying to commit. It is technically possible to do this currently, but upon completion of #3320 it would make sense to refactor `_RewriteFiles` to utilize the `_validate_concurrency()` function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-6103967972
Bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3089966996
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +667,96 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValueError("Cannot delete files that are not present in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
+ raise ValueError(f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})")
+
+ return super()._commit()
+ else:
+ return (), ()
+
+ def _deleted_entries(self) -> list[ManifestEntry]:
+ """Check if we need to mark the files as deleted."""
+ if self._parent_snapshot_id is not None:
+ previous_snapshot = self._transaction.table_metadata.snapshot_by_id(self._parent_snapshot_id)
+ if previous_snapshot is None:
+ raise ValueError(f"Could not find the previous snapshot: {self._parent_snapshot_id}")
+
+ executor = ExecutorFactory.get_or_create()
+
+ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
+ return [
+ ManifestEntry.from_args(
+ status=ManifestEntryStatus.DELETED,
+ snapshot_id=self.snapshot_id,
+ sequence_number=entry.sequence_number,
+ file_sequence_number=entry.file_sequence_number,
+ data_file=entry.data_file,
+ )
+ for entry in manifest.fetch_manifest_entry(self._io, discard_deleted=True)
+ if entry.data_file.content == DataFileContent.DATA and entry.data_file in self._deleted_data_files
+ ]
+
+ list_of_entries = executor.map(_get_entries, previous_snapshot.manifests(self._io))
+ return list(itertools.chain(*list_of_entries))
+ else:
+ return []
+
+ def _existing_manifests(self) -> list[ManifestFile]:
+ """To determine if there are any existing manifests."""
Review Comment:
Hi @geruh, thank you for the feedback! I agree with this idea that the two sets of code are highly similar, in b0a770c I create a `_get_existing_manifests` function in `_SnapshotProducer` that is reused in both `_OverwriteFiles` and `_RewriteFiles`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-4146501702
Hi @kevinjqliu, apologies for the delay, thank you so much for taking the time to review the PR again, I understand that you are quite busy. I've addressed all your points in the latest set of tests within 33aaef0817b1366d80bcd8194a0c2ca5ba6f46f2. I've thoroughly expanded the tests to integrate those requirements across a broad set of tests.
There are two minor issues I noticed however:
- Requirement: If the difference is due to prior soft-deletes, confirm those delete files account for it
- This would require however that the `_RewriteFiles` be scoped to handle Delete Manifests, but currently it's only set to handle Data Files. Handling Delete Manifests would make it so that we could potentially do `REPLACE` operations on deleted files. For example the purpose of this PR is to allow for compaction of data files, but we could in theory also compact delete files for the use case that someone has run many delete operations on many small files.
- I think this is definitely something to work on, but perhaps not in this PR. The Java code seems to handle this well. I am thinking that after we merge this `REPLACE`, we can next work on the data compaction issue. Then after that we can come back to work on _RewriteFiles for Delete Manifests and work on metadata compaction afterwards.
- Another more minor issue I noticed is that from running the tests and doing `fast_append()` on files that are `DataFileContent.POSITION_DELETES`, they're not yet being labeled properly as `ManifestContent.DELETES`. IIUC this is due to the fact that `_SnapshotProducer._manifests()` (which `fast_append` relies on under the hood) currently defaults to creating standard `ManifestContent.DATA` writers. It doesn't yet inspect the incoming file's content type to route `POSITION_DELETES` into a dedicated `ManifestContent.DELETES` writer. I worked around this in my test by scanning the manifest entries directly rather than relying on the manifest's label, but I just wanted to flag it for the roadmap for when we build out full Merge-on-Read write support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3983953126
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -852,6 +850,120 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files.
+
+ Produces a REPLACE snapshot that swaps existing data files for new ones without
+ changing the logical contents of the table. This is the metadata-only operation
+ used by compaction (bin-packing, sort, format migration).
+
+ Current scope:
+ - Data file rewriting only (delete + add DataFiles)
+ - Validates: files-to-delete exist, added_records <= deleted_records,
+ no new delete files conflict with replaced data files
+
+ Future work (additive — no structural changes needed):
+ - Delete-file rewriting (add _deleted_delete_files set + separate manifest handling)
+ - dataSequenceNumber override (pin new files' seq to match replaced, for eq-delete safety)
+ - validateFromSnapshot (expose _starting_snapshot_id setter for long-running planners)
+ - ignoreEqualityDeletes in validation (coupled with dataSequenceNumber)
+ """
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValidationException("Cannot commit, missing data files to be rewritten that are not in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
+ raise ValidationException(
+ f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})"
+ )
+
+ return super()._commit()
+ else:
+ return (), ()
+
+ @cached_property
+ def _cached_deleted_entries(self) -> list[ManifestEntry]:
Review Comment:
Done, `_cached_deleted_entries` and the executor/manifest-scanning boilerplate now live in `_SnapshotProducer`. Each subclass just overrides `_get_deleted_manifest_entries(manifest)` to control which entries get marked as `DELETED`. `_OverwriteFiles` adds partition-level evaluator pruning in its override and `_RewriteFiles` matches on `_deleted_data_files` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3983944830
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -852,6 +850,120 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files.
+
+ Produces a REPLACE snapshot that swaps existing data files for new ones without
+ changing the logical contents of the table. This is the metadata-only operation
+ used by compaction (bin-packing, sort, format migration).
+
+ Current scope:
+ - Data file rewriting only (delete + add DataFiles)
+ - Validates: files-to-delete exist, added_records <= deleted_records,
+ no new delete files conflict with replaced data files
+
+ Future work (additive — no structural changes needed):
+ - Delete-file rewriting (add _deleted_delete_files set + separate manifest handling)
+ - dataSequenceNumber override (pin new files' seq to match replaced, for eq-delete safety)
+ - validateFromSnapshot (expose _starting_snapshot_id setter for long-running planners)
+ - ignoreEqualityDeletes in validation (coupled with dataSequenceNumber)
+ """
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
Review Comment:
Thanks @rambleraptor, I've flattened `_commit()` into guard-clause style: the no-op early return is now at the top, and the rest is a flat sequence of `if invariant -> raise` checks ending with `super()._commit()`. I think the code suggestion has a small logic flip (`not deleted and added` vs `not deleted and not added`) but the structural feedback is addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub) via issues" <is...@iceberg.apache.org>.
qzyu999 commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-6810588645
bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3131: feat: Add metadata-only replace API to Table for REPLACE snapshot operations
URL: https://github.com/apache/iceberg-python/pull/3131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3096467745
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +669,70 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValueError("Cannot delete files that are not present in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
+ raise ValueError(f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})")
+
+ return super()._commit()
+ else:
+ return (), ()
+
+ @cached_property
+ def _cached_deleted_entries(self) -> list[ManifestEntry]:
+ """Check if we need to mark the files as deleted."""
+ if self._parent_snapshot_id is not None:
+ previous_snapshot = self._transaction.table_metadata.snapshot_by_id(self._parent_snapshot_id)
+ if previous_snapshot is None:
+ raise ValueError(f"Could not find the previous snapshot: {self._parent_snapshot_id}")
+
+ executor = ExecutorFactory.get_or_create()
+
+ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
+ return [
+ ManifestEntry.from_args(
+ status=ManifestEntryStatus.DELETED,
+ snapshot_id=self.snapshot_id,
+ sequence_number=entry.sequence_number,
+ file_sequence_number=entry.file_sequence_number,
+ data_file=entry.data_file,
+ )
+ for entry in manifest.fetch_manifest_entry(self._io, discard_deleted=True)
+ if entry.data_file.content == DataFileContent.DATA and entry.data_file in self._deleted_data_files
+ ]
+
+ list_of_entries = executor.map(_get_entries, previous_snapshot.manifests(self._io))
+ return list(itertools.chain(*list_of_entries))
+ else:
+ return []
+
+ def _deleted_entries(self) -> list[ManifestEntry]:
+ """Check if we need to mark the files as deleted."""
+ return self._cached_deleted_entries
+
+ def _existing_manifests(self) -> list[ManifestFile]:
+ """To determine if there are any existing manifests."""
+ return self._get_existing_manifests()
Review Comment:
Hi @geruh, thanks for pointing this out, these doc strings in `_RewriteFiles` have been removed in c60d5ad62511b5db8d8f7e1eca98af50e2aee3f4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3090686330
##########
tests/table/test_replace.py:
##########
@@ -0,0 +1,458 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import cast
+
+import pytest
+
+from pyiceberg.catalog import Catalog
+from pyiceberg.manifest import (
+ DataFile,
+ DataFileContent,
+ FileFormat,
+ ManifestEntry,
+ ManifestEntryStatus,
+)
+from pyiceberg.schema import Schema
+from pyiceberg.table.snapshots import Operation, Snapshot, Summary
+from pyiceberg.typedef import Record
+
+
+def test_replace_internally(catalog: Catalog) -> None:
Review Comment:
Hi @geruh, these are all great test ideas, I've added them in d939b67.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3983948055
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -852,6 +850,120 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files.
+
+ Produces a REPLACE snapshot that swaps existing data files for new ones without
+ changing the logical contents of the table. This is the metadata-only operation
+ used by compaction (bin-packing, sort, format migration).
+
+ Current scope:
+ - Data file rewriting only (delete + add DataFiles)
+ - Validates: files-to-delete exist, added_records <= deleted_records,
+ no new delete files conflict with replaced data files
+
+ Future work (additive — no structural changes needed):
+ - Delete-file rewriting (add _deleted_delete_files set + separate manifest handling)
+ - dataSequenceNumber override (pin new files' seq to match replaced, for eq-delete safety)
+ - validateFromSnapshot (expose _starting_snapshot_id setter for long-running planners)
+ - ignoreEqualityDeletes in validation (coupled with dataSequenceNumber)
+ """
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValidationException("Cannot commit, missing data files to be rewritten that are not in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
+ raise ValidationException(
+ f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})"
+ )
+
+ return super()._commit()
+ else:
+ return (), ()
+
+ @cached_property
+ def _cached_deleted_entries(self) -> list[ManifestEntry]:
+ """Build manifest entries marking deleted data files with DELETED status."""
+ if self._parent_snapshot_id is not None:
+ previous_snapshot = self._transaction.table_metadata.snapshot_by_id(self._parent_snapshot_id)
+ if previous_snapshot is None:
+ raise ValueError(f"Could not find the previous snapshot: {self._parent_snapshot_id}")
+
+ executor = ExecutorFactory.get_or_create()
+
+ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
+ return [
+ ManifestEntry.from_args(
+ status=ManifestEntryStatus.DELETED,
+ snapshot_id=self.snapshot_id,
+ sequence_number=entry.sequence_number,
+ file_sequence_number=entry.file_sequence_number,
+ data_file=entry.data_file,
+ )
+ for entry in manifest.fetch_manifest_entry(self._io, discard_deleted=True)
+ if entry.data_file.content == DataFileContent.DATA and entry.data_file in self._deleted_data_files
+ ]
+
+ list_of_entries = executor.map(_get_entries, previous_snapshot.manifests(self._io))
+ return list(itertools.chain(*list_of_entries))
+ else:
+ return []
+
+ def _deleted_entries(self) -> list[ManifestEntry]:
+ return self._cached_deleted_entries
+
+ def _existing_manifests(self) -> list[ManifestFile]:
+ return self._get_existing_manifests()
Review Comment:
Done, `should_use_manifest_pruning` is now a required parameter with no default.` _OverwriteFiles` passes `True`, `_RewriteFiles` passes `False`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-6810522993
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r2991623781
##########
pyiceberg/table/__init__.py:
##########
@@ -450,6 +450,32 @@ def update_statistics(self) -> UpdateStatistics:
"""
return UpdateStatistics(transaction=self)
+ def replace(
Review Comment:
we should not expose `replace` as a public function as we cannot guarantee that the `files_to_delete` and `files_to_add` contains the same records.
I think we should start at `_RewriteFiles`
##########
tests/table/test_replace.py:
##########
@@ -0,0 +1,109 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from pyiceberg.catalog import Catalog
+from pyiceberg.manifest import DataFile, DataFileContent, FileFormat
+from pyiceberg.schema import Schema
+from pyiceberg.table.snapshots import Operation
+from pyiceberg.typedef import Record
+
+
+def test_replace_api(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace",
+ schema=Schema(),
+ )
+
+ # Create mock DataFiles for the test
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ file_to_add = DataFile.from_args(
+ file_path="s3://bucket/test/data/added.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_add.spec_id = 0
+
+ # Initially append to have something to replace
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+
+ # Verify initial append snapshot
+ assert len(table.history()) == 1
+ snapshot = table.current_snapshot()
+ assert snapshot is not None
+ assert snapshot.summary is not None
+ assert snapshot.summary["operation"] == Operation.APPEND
+
+ # Call the replace API
+ table.replace(files_to_delete=[file_to_delete], files_to_add=[file_to_add])
+
+ # Verify the replacement created a REPLACE snapshot
+ assert len(table.history()) == 2
+ snapshot = table.current_snapshot()
+ assert snapshot is not None
+ assert snapshot.summary is not None
+ assert snapshot.summary["operation"] == Operation.REPLACE
+
+ # Verify the correct files are added and deleted
+ # The summary property tracks these counts
+ assert snapshot.summary["added-data-files"] == "1"
+ assert snapshot.summary["deleted-data-files"] == "1"
+ assert snapshot.summary["added-records"] == "100"
+ assert snapshot.summary["deleted-records"] == "100"
+
+ # Verify the new file exists in the new manifest
+ manifest_files = snapshot.manifests(table.io)
+ assert len(manifest_files) == 2 # One for ADDED, one for DELETED
+
+ # Check that sequence numbers were handled properly natively by verifying the manifest contents
+ entries = []
+ for manifest in manifest_files:
+ for entry in manifest.fetch_manifest_entry(table.io, discard_deleted=False):
+ entries.append(entry)
+
+ # One entry for ADDED (new file), one for DELETED (old file)
+ assert len(entries) == 2
Review Comment:
we need to test a bit more on the status of each file.
https://github.com/apache/iceberg/blob/main/core/src/test/java/org/apache/iceberg/TestRewriteFiles.java is a good reference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3088999476
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +667,96 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValueError("Cannot delete files that are not present in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
+ raise ValueError(f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})")
+
+ return super()._commit()
+ else:
+ return (), ()
+
+ def _deleted_entries(self) -> list[ManifestEntry]:
+ """Check if we need to mark the files as deleted."""
+ if self._parent_snapshot_id is not None:
+ previous_snapshot = self._transaction.table_metadata.snapshot_by_id(self._parent_snapshot_id)
+ if previous_snapshot is None:
+ raise ValueError(f"Could not find the previous snapshot: {self._parent_snapshot_id}")
+
+ executor = ExecutorFactory.get_or_create()
+
+ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
+ return [
+ ManifestEntry.from_args(
+ status=ManifestEntryStatus.DELETED,
+ snapshot_id=self.snapshot_id,
+ sequence_number=entry.sequence_number,
+ file_sequence_number=entry.file_sequence_number,
+ data_file=entry.data_file,
+ )
+ for entry in manifest.fetch_manifest_entry(self._io, discard_deleted=True)
+ if entry.data_file.content == DataFileContent.DATA and entry.data_file in self._deleted_data_files
+ ]
+
+ list_of_entries = executor.map(_get_entries, previous_snapshot.manifests(self._io))
+ return list(itertools.chain(*list_of_entries))
+ else:
+ return []
+
+ def _existing_manifests(self) -> list[ManifestFile]:
+ """To determine if there are any existing manifests."""
Review Comment:
This logic looks nearly identical to the `OverwriteFiles._existing_manifests()`, and probably can use some clean python oop with the different snapshot classes. So let's create a helper in the `_SnapshotProducer` class. Then Overwrite can add it's additional logic on top and rewrite can just use it.
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +667,96 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
Review Comment:
I think we can replicate the `_DeleteFiles` logic here by using the `@cache_property` on _the compute deletes function. Especially since `_commit()` calls `self._deleted_entries()` for validation and then calls the super commit to write and get delete entries.
https://github.com/apache/iceberg-python/blob/721c5aa504e2c2069a197f01a238070e37e389d8/pyiceberg/table/update/snapshot.py#L407
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -667,6 +667,96 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files."""
+
+ def __init__(self, operation: Operation, transaction: Transaction, io: FileIO, snapshot_properties: dict[str, str]):
+ super().__init__(operation, transaction, io, snapshot_properties=snapshot_properties)
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValueError("Cannot delete files that are not present in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
Review Comment:
Where are you seeing this invariant? I mean this seems correct since the spec says rewrite must be "logically equivalent". This check could reasonable as a safety guard, but what happens when delete file rewriting is added? Then these numbers could be incorrect.
##########
tests/table/test_replace.py:
##########
@@ -0,0 +1,458 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import cast
+
+import pytest
+
+from pyiceberg.catalog import Catalog
+from pyiceberg.manifest import (
+ DataFile,
+ DataFileContent,
+ FileFormat,
+ ManifestEntry,
+ ManifestEntryStatus,
+)
+from pyiceberg.schema import Schema
+from pyiceberg.table.snapshots import Operation, Snapshot, Summary
+from pyiceberg.typedef import Record
+
+
+def test_replace_internally(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace",
+ schema=Schema(),
+ )
+
+ # 1. File we will delete
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ # 2. File we will leave completely untouched
+ file_to_keep = DataFile.from_args(
+ file_path="s3://bucket/test/data/kept.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=50,
+ file_size_in_bytes=512,
+ content=DataFileContent.DATA,
+ )
+ file_to_keep.spec_id = 0
+
+ # 3. File we are adding as a replacement
+ file_to_add = DataFile.from_args(
+ file_path="s3://bucket/test/data/added.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_add.spec_id = 0
+
+ # Initially append BOTH the file to delete and the file to keep
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+ append_snapshot.append_data_file(file_to_keep)
+
+ old_snapshot = cast(Snapshot, table.current_snapshot())
+ old_snapshot_id = old_snapshot.snapshot_id
+ old_sequence_number = cast(int, old_snapshot.sequence_number)
+
+ # Call the internal replace API
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_to_delete)
+ rewrite.append_data_file(file_to_add)
+
+ snapshot = cast(Snapshot, table.current_snapshot())
+ summary = cast(Summary, snapshot.summary)
+
+ # 1. Has a unique snapshot ID
+ assert snapshot.snapshot_id is not None
+ assert snapshot.snapshot_id != old_snapshot_id
+
+ # 2. Parent points to the previous snapshot
+ assert snapshot.parent_snapshot_id == old_snapshot_id
+
+ # 3. Sequence number is exactly previous + 1
+ assert snapshot.sequence_number == old_sequence_number + 1
+
+ # 4. Operation type is set to "replace"
+ assert summary["operation"] == Operation.REPLACE
+
+ # 5. Manifest list path is correct (just verify it exists and is a string path)
+ assert snapshot.manifest_list is not None
+ assert isinstance(snapshot.manifest_list, str)
+
+ # 6. Summary counts are accurate
+ assert summary["added-data-files"] == "1"
+ assert summary["deleted-data-files"] == "1"
+ assert summary["added-records"] == "100"
+ assert summary["deleted-records"] == "100"
+ assert summary["total-records"] == "150"
+
+ # Fetch all entries from the new manifests
+ manifest_files = snapshot.manifests(table.io)
+ entries: list[ManifestEntry] = []
+ for manifest in manifest_files:
+ entries.extend(manifest.fetch_manifest_entry(table.io, discard_deleted=False))
+
+ # We expect 3 entries: ADDED, DELETED, and EXISTING
+ assert len(entries) == 3
+
+ # Check ADDED
+ added_entries = [e for e in entries if e.status == ManifestEntryStatus.ADDED]
+ assert len(added_entries) == 1
+ assert added_entries[0].data_file.file_path == file_to_add.file_path
+ assert added_entries[0].snapshot_id == snapshot.snapshot_id
+
+ # Check DELETED
+ deleted_entries = [e for e in entries if e.status == ManifestEntryStatus.DELETED]
+ assert len(deleted_entries) == 1
+ assert deleted_entries[0].data_file.file_path == file_to_delete.file_path
+ assert deleted_entries[0].snapshot_id == snapshot.snapshot_id
+
+ # Check EXISTING
+ existing_entries = [e for e in entries if e.status == ManifestEntryStatus.EXISTING]
+ assert len(existing_entries) == 1
+ assert existing_entries[0].data_file.file_path == file_to_keep.file_path
+ assert existing_entries[0].snapshot_id == old_snapshot_id
+
+
+def test_replace_reuses_unaffected_manifests(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_reuse_manifest",
+ schema=Schema(),
+ )
+
+ file_a = DataFile.from_args(
+ file_path="s3://bucket/test/data/a.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_a.spec_id = 0
+
+ file_b = DataFile.from_args(
+ file_path="s3://bucket/test/data/b.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_b.spec_id = 0
+
+ file_c = DataFile.from_args(
+ file_path="s3://bucket/test/data/c.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_c.spec_id = 0
+
+ # Commit 1: Append file A (Creates Manifest 1)
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_a)
+
+ # Commit 2: Append file B (Creates Manifest 2)
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_b)
+
+ snapshot_before = cast(Snapshot, table.current_snapshot())
+ manifests_before = snapshot_before.manifests(table.io)
+ assert len(manifests_before) == 2
+
+ # Identify which manifest belongs to file_b and file_a
+ manifest_b_path = None
+ manifest_a_path = None
+ for m in manifests_before:
+ entries = m.fetch_manifest_entry(table.io, discard_deleted=False)
+ if any(e.data_file.file_path == file_b.file_path for e in entries):
+ manifest_b_path = m.manifest_path
+ if any(e.data_file.file_path == file_a.file_path for e in entries):
+ manifest_a_path = m.manifest_path
+
+ assert manifest_b_path is not None
+ assert manifest_a_path is not None
+
+ # Commit 3: Replace file A with file C
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_a)
+ rewrite.append_data_file(file_c)
+
+ snapshot_after = cast(Snapshot, table.current_snapshot())
+ assert snapshot_after is not None
+ manifests_after = snapshot_after.manifests(table.io)
+
+ # We expect 3 manifests:
+ # 1. The reused one for file B
+ # 2. The newly rewritten one marking file A as DELETED
+ # 3. The new one for file C (ADDED)
+ assert len(manifests_after) == 3
+
+ manifest_paths_after = [m.manifest_path for m in manifests_after]
+
+ # ASSERTION 1: The untouched manifest is completely reused (the path matches exactly)
+ assert manifest_b_path in manifest_paths_after
+
+ # ASSERTION 2: File A's old manifest is NOT reused (since it was rewritten to change status to DELETED)
+ assert manifest_a_path not in manifest_paths_after
+
+
+def test_replace_empty_files(catalog: Catalog) -> None:
+ # Setup a basic table using the catalog fixture
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_empty",
+ schema=Schema(),
+ )
+
+ # Replacing empty lists should not throw errors, but should produce no changes.
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace():
+ pass # Entering and exiting the context manager without adding/deleting
+
+ # History should be completely empty since no files were rewritten
+ assert len(table.history()) == 0
+ assert table.current_snapshot() is None
+
+
+def test_replace_missing_file_abort(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_missing",
+ schema=Schema(),
+ )
+
+ fake_data_file = DataFile.from_args(
+ file_path="s3://bucket/test/data/does_not_exist.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ fake_data_file.spec_id = 0
+
+ new_data_file = DataFile.from_args(
+ file_path="s3://bucket/test/data/new.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ new_data_file.spec_id = 0
+
+ # Ensure it aborts when trying to replace a file that isn't in the table
+ with pytest.raises(ValueError, match="Cannot delete files that are not present in the table"):
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(fake_data_file)
+ rewrite.append_data_file(new_data_file)
+
+
+def test_replace_invariant_violation(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_invariant",
+ schema=Schema(),
+ )
+
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ # Create a new file with MORE records than the one we are deleting
+ too_many_records_file = DataFile.from_args(
+ file_path="s3://bucket/test/data/too_many.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=101,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ too_many_records_file.spec_id = 0
+
+ # Initially append to have something to replace
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+
+ # Ensure it enforces the invariant: records added <= records removed
+ with pytest.raises(ValueError, match=r"Invalid replace: records added \(101\) exceeds records removed \(100\)"):
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_to_delete)
+ rewrite.append_data_file(too_many_records_file)
+
+
+def test_replace_allows_shrinking_for_soft_deletes(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_shrink",
+ schema=Schema(),
+ )
+
+ # Old data file has 100 records
+ file_to_delete = DataFile.from_args(
+ file_path="s3://bucket/test/data/deleted.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=100,
+ file_size_in_bytes=1024,
+ content=DataFileContent.DATA,
+ )
+ file_to_delete.spec_id = 0
+
+ # New data file only has 90 records (simulating 10 records were soft-deleted)
+ shrunk_file_to_add = DataFile.from_args(
+ file_path="s3://bucket/test/data/shrunk.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=90,
+ file_size_in_bytes=900,
+ content=DataFileContent.DATA,
+ )
+ shrunk_file_to_add.spec_id = 0
+
+ # Initially append
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_to_delete)
+
+ # This should succeed without throwing an invariant violation
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_to_delete)
+ rewrite.append_data_file(shrunk_file_to_add)
+
+ snapshot = cast(Snapshot, table.current_snapshot())
+ summary = cast(Summary, snapshot.summary)
+
+ assert summary["operation"] == Operation.REPLACE
+ assert summary["added-records"] == "90"
+ assert summary["deleted-records"] == "100"
+
+
+def test_replace_passes_through_delete_manifests(catalog: Catalog) -> None:
+ # Setup a basic table
+ catalog.create_namespace("default")
+ table = catalog.create_table(
+ identifier="default.test_replace_delete_manifests",
+ schema=Schema(),
+ properties={"format-version": "2"},
+ )
+
+ # 1. Data file we will replace
+ file_a = DataFile.from_args(
+ file_path="s3://bucket/test/data/a.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_a.spec_id = 0
+
+ # 2. A Position Delete file (representing row-level deletes)
+ file_a_deletes = DataFile.from_args(
+ file_path="s3://bucket/test/data/a_deletes.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=2,
+ file_size_in_bytes=50,
+ content=DataFileContent.POSITION_DELETES,
+ )
+ file_a_deletes.spec_id = 0
+
+ # 3. Data file we are adding as a replacement
+ file_b = DataFile.from_args(
+ file_path="s3://bucket/test/data/b.parquet",
+ file_format=FileFormat.PARQUET,
+ partition=Record(),
+ record_count=10,
+ file_size_in_bytes=100,
+ content=DataFileContent.DATA,
+ )
+ file_b.spec_id = 0
+
+ # Commit 1: Append the data file
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_a)
+
+ # Commit 2: Append the delete file
+ with table.transaction() as tx:
+ with tx.update_snapshot().fast_append() as append_snapshot:
+ append_snapshot.append_data_file(file_a_deletes)
+
+ # Find the path of the delete manifest so we can verify it survives
+ snapshot_before = cast(Snapshot, table.current_snapshot())
+ manifests_before = snapshot_before.manifests(table.io)
+
+ delete_manifest_path = None
+ for m in manifests_before:
+ entries = m.fetch_manifest_entry(table.io, discard_deleted=False)
+ if any(e.data_file.file_path == file_a_deletes.file_path for e in entries):
+ delete_manifest_path = m.manifest_path
+ break
+
+ assert delete_manifest_path is not None
+
+ # Commit 3: Replace data file A with data file B
+ with table.transaction() as tx:
+ with tx.update_snapshot().replace() as rewrite:
+ rewrite.delete_data_file(file_a)
+ rewrite.append_data_file(file_b)
+
+ # Verify the delete manifest was passed through unchanged
+ snapshot_after = cast(Snapshot, table.current_snapshot())
+ assert snapshot_after is not None
+ manifests_after = snapshot_after.manifests(table.io)
+ manifest_paths_after = [m.manifest_path for m in manifests_after]
+
+ assert delete_manifest_path in manifest_paths_after
Review Comment:
We should also test to ensure that original sequence numbers are carried over from before the rewrite.
##########
tests/table/test_replace.py:
##########
@@ -0,0 +1,458 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from typing import cast
+
+import pytest
+
+from pyiceberg.catalog import Catalog
+from pyiceberg.manifest import (
+ DataFile,
+ DataFileContent,
+ FileFormat,
+ ManifestEntry,
+ ManifestEntryStatus,
+)
+from pyiceberg.schema import Schema
+from pyiceberg.table.snapshots import Operation, Snapshot, Summary
+from pyiceberg.typedef import Record
+
+
+def test_replace_internally(catalog: Catalog) -> None:
Review Comment:
Tests are on the right track can we add a few more like:
- test with multiple files for set matching rows
- test against partitioned table
- did we do a no-op replace to ensure we have correct state matching java impl
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -724,6 +814,14 @@ def delete(self) -> _DeleteFiles:
snapshot_properties=self._snapshot_properties,
)
+ def replace(self) -> _RewriteFiles:
+ return _RewriteFiles(
Review Comment:
I'm sort of confused by the naming since we are introducing a user facing API `replace` but the underlying snapshot operation is a rewrite? We should rename to rewrite for consistency? Unless I'm missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3090633495
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -724,6 +814,14 @@ def delete(self) -> _DeleteFiles:
snapshot_properties=self._snapshot_properties,
)
+ def replace(self) -> _RewriteFiles:
+ return _RewriteFiles(
Review Comment:
Hi @geruh, you bring up a good point, and it's something I noticed seemed off along the way. The reason why we have this discrepancy is because we're mirroring what's found in the Java code itself.
- Looking here (https://iceberg.apache.org/javadoc/latest/org/apache/iceberg/RewriteFiles.html), you can see that the "API for replacing files in a table." is called `RewriteFiles`.
- Looking here (https://iceberg.apache.org/javadoc/1.4.2/org/apache/iceberg/DataOperations.html#REPLACE), you can see that the `REPLACE` operation is implemented by `RewriteFiles`.
- On a slightly unrelated note here (https://iceberg.apache.org/javadoc/1.4.2/org/apache/iceberg/DataOperations.html#OVERWRITE), the `OVERWRITE` operation itself can be implemented by `ReplacePartitions`.
I named the Python API `replace()` to accurately reflect the `Operation.REPLACE` snapshot string it generates, while keeping the internal class named `_RewriteFiles` to match the Java builder logic.
That said, if you feel strongly about matching the Java API's user-facing method (`rewrite()`) rather than the snapshot operation (`replace()`), I'm happy to rename the public method to `rewrite()` for consistency. Let me know what you prefer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-4713895076
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#discussion_r3983953126
##########
pyiceberg/table/update/snapshot.py:
##########
@@ -852,6 +850,120 @@ def _get_entries(manifest: ManifestFile) -> list[ManifestEntry]:
return []
+class _RewriteFiles(_SnapshotProducer["_RewriteFiles"]):
+ """A snapshot producer that rewrites data files.
+
+ Produces a REPLACE snapshot that swaps existing data files for new ones without
+ changing the logical contents of the table. This is the metadata-only operation
+ used by compaction (bin-packing, sort, format migration).
+
+ Current scope:
+ - Data file rewriting only (delete + add DataFiles)
+ - Validates: files-to-delete exist, added_records <= deleted_records,
+ no new delete files conflict with replaced data files
+
+ Future work (additive — no structural changes needed):
+ - Delete-file rewriting (add _deleted_delete_files set + separate manifest handling)
+ - dataSequenceNumber override (pin new files' seq to match replaced, for eq-delete safety)
+ - validateFromSnapshot (expose _starting_snapshot_id setter for long-running planners)
+ - ignoreEqualityDeletes in validation (coupled with dataSequenceNumber)
+ """
+
+ def _commit(self) -> UpdatesAndRequirements:
+ # Only produce a commit when there is something to rewrite
+ if self._deleted_data_files or self._added_data_files:
+ # Grab the entries that we actually found in the table's manifests
+ deleted_entries = self._deleted_entries()
+ found_deleted_files = {entry.data_file for entry in deleted_entries}
+
+ # If the user asked to delete files that aren't in the table, abort.
+ if len(found_deleted_files) != len(self._deleted_data_files):
+ raise ValidationException("Cannot commit, missing data files to be rewritten that are not in the table")
+
+ added_records = sum(f.record_count for f in self._added_data_files)
+ deleted_records = sum(entry.data_file.record_count for entry in deleted_entries)
+
+ if added_records > deleted_records:
+ raise ValidationException(
+ f"Invalid replace: records added ({added_records}) exceeds records removed ({deleted_records})"
+ )
+
+ return super()._commit()
+ else:
+ return (), ()
+
+ @cached_property
+ def _cached_deleted_entries(self) -> list[ManifestEntry]:
Review Comment:
Done, `_cached_deleted_entries` and the executor/manifest-scanning boilerplate now live in `_SnapshotProducer`. Each subclass just overrides `_get_deleted_manifest_entries(manifest)` to control which entries get marked as `DELETED`. `_OverwriteFiles` adds partition-level evaluator pruning in its override; `_RewriteFiles` matches on `_deleted_data_files` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add metadata-only replace API to Table for REPLACE snapshot operations [iceberg-python]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on PR #3131:
URL: https://github.com/apache/iceberg-python/pull/3131#issuecomment-5337222247
Hi @kevinjqliu, @geruh, thanks for reopening the PR. Rebased and updated now that #3320 has merged. Here are the latest changes:
**Integration with commit retry/validation (#3320):**
- Added `_validate_concurrency()` override on `_RewriteFiles` that matches Java's `BaseRewriteFiles.validate()`, only checks `validateNoNewDeletesForDataFiles`, does NOT reject concurrent appends (those are not conflicts for a replace)
- Added `_refresh_for_retry()` that clears the `@cached_property` so manifest entries are recomputed against the refreshed parent snapshot on retry
CC: @lawofcycles, congrats on completing #3320, PTAL if you have time for these changes here. I believe that I've built in accordance with what we had discussed previously.
**Other changes since last review:**
- Refactored `_existing_manifests` into a shared `_get_existing_manifests()` method on `_SnapshotProducer` (used by both `_OverwriteFiles` and `_RewriteFiles`)
- Changed validation errors from `ValueError` to `ValidationException` for consistency with the rest of the validation infrastructure
**Test coverage (18 tests, all passing across memory/sql/sql_without_rowcount):**
- Core replace: snapshot metadata, summary counts, manifest entry status/sequence numbers
- Manifest reuse for unaffected files
- Delete-manifest passthrough (v2)
- No-op on empty input (both empty and non-empty tables)
- Missing-file abort (`ValidationException`)
- Records invariant violation (`added > deleted` -> `ValidationException`)
- Soft-delete shrinking (`added < deleted` -> success)
- Multi-file, partitioned table, custom branch
- Concurrent non-conflicting append in different partition -> retry succeeds
- Concurrent replace on different partitions -> both land
- Concurrent append to same unpartitioned table -> NOT a conflict (key Java-parity behavior)
- Concurrent delete of same file -> `ValidationException`
- `_refresh_for_retry` clears cached state
- Add-only replace raises invariant violation
**Scoped to data-file rewriting only.** Delete-file rewriting, `dataSequenceNumber` override, and `validateFromSnapshot` are documented as future additive extensions in the class docstring. These exist in the Java version and should be readily extendable without major changes to the existing code structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org