You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/08/19 05:36:18 UTC

[PR] Validate replaced files during commit retries [iceberg-python]

kevinjqliu opened a new pull request, #3811:
URL: https://github.com/apache/iceberg-python/pull/3811

   ## Summary
   
   - validate that explicitly replaced data files were not concurrently deleted
   - preserve successful retries when another file in the same or a different partition is deleted
   - add regression coverage for all three concurrency outcomes
   
   ## Rationale
   
   Follow-up to #3780. That change correctly moved explicit file partition matching out of the row-predicate projection path. For a low-level file overwrite, however, the row predicate can remain `AlwaysFalse`, so the existing predicate-based deleted-data-file validation does not run.
   
   If another transaction deletes the target file before the overwrite retries, the stale replacement could otherwise be appended and resurrect deleted data. This change validates the exact replaced files against concurrent data-file deletions. Partition records narrow the manifest scan, followed by exact file-path matching to avoid conflicts with unrelated files.
   
   ## Tests
   
   - `uv run --all-extras python -m pytest -q tests/table/test_commit_retry.py tests/table/test_validate.py tests/table/test_upsert.py tests/table/test_init.py::test_build_large_partition_predicate tests/table/test_init.py::test_overwrite_delete_data_file_on_bucket_partition` — 166 passed
   - ruff, ruff-format, mypy, and pydocstyle hooks passed for all changed files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Validate replaced data files during commit retries [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #3811:
URL: https://github.com/apache/iceberg-python/pull/3811#issuecomment-5344269275

   @lawofcycles fyi 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Validate replaced data files during commit retries [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #3811:
URL: https://github.com/apache/iceberg-python/pull/3811#issuecomment-5346741431

   giving some more time for feedback before i merge this. 
   
   fyi @rambleraptor this should be the last issue related to #3758 and #3779 for the 0.12 release 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Validate replaced data files during commit retries [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #3811:
URL: https://github.com/apache/iceberg-python/pull/3811#discussion_r3817931227


##########
pyiceberg/table/update/snapshot.py:
##########


Review Comment:
   sorry i didnt understand this comment about outside the commit window. i think for retries we only need to look at the commit window defined



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Validate replaced data files during commit retries [iceberg-python]

Posted by "rambleraptor (via GitHub)" <gi...@apache.org>.
rambleraptor commented on code in PR #3811:
URL: https://github.com/apache/iceberg-python/pull/3811#discussion_r3816138067


##########
pyiceberg/table/update/validate.py:
##########
@@ -195,6 +195,33 @@ def _validate_deleted_data_files(
         raise ValidationException(f"Deleted data files were found matching the filter for snapshots {conflicting_snapshots}!")
 
 
+def _validate_data_files_exist(
+    table: Table,
+    starting_snapshot: Snapshot,
+    data_files: set[DataFile],
+    parent_snapshot: Snapshot | None,
+) -> None:
+    """Validate that explicitly replaced data files have not been concurrently deleted.
+
+    Args:
+        table: Table to validate
+        starting_snapshot: Snapshot at the end of the validation window
+        data_files: Data files that must still exist
+        parent_snapshot: Snapshot at the start of the validation window, excluded from the scan
+    """
+    partition_set: dict[int, set[Record]] = {}

Review Comment:
   nit: You can use a `collections.defaultdict` to avoid having to setup the default sets.



##########
pyiceberg/table/update/snapshot.py:
##########


Review Comment:
   It looks like this new method won't catch anything outside of the commit window. Java still looks through everything (which may be wasteful).
   
   I say we punt that to a follow-up PR just so we can unblock the release.



##########
tests/table/test_commit_retry.py:
##########
@@ -323,6 +323,79 @@ def test_concurrent_overwrite_overwrite_raises_validation_exception(catalog: Cat
         tbl2.overwrite(pa.table({"x": [40, 50, 60]}), overwrite_filter="x > 0")
 
 
+@pytest.mark.parametrize(
+    ("concurrently_deleted_file", "expect_conflict", "expected_values"),
+    [
+        pytest.param("target", True, [1, 3], id="target-file"),
+        pytest.param("same-partition", False, [1, 2], id="same-partition-file"),
+        pytest.param("different-partition", False, [2, 3], id="different-partition-file"),
+    ],
+)
+def test_file_overwrite_validates_concurrent_file_delete(
+    catalog: Catalog,
+    concurrently_deleted_file: str,
+    expect_conflict: bool,
+    expected_values: list[int],
+) -> None:
+    """A file replacement must fail only when its target file was concurrently deleted."""
+    import uuid

Review Comment:
   nit: Since this is in the stdlib, we can probably put it on the normal imports up top.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Validate replaced data files during commit retries [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #3811:
URL: https://github.com/apache/iceberg-python/pull/3811#issuecomment-5359164675

   thanks for the reviews @rambleraptor @geruh 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Validate replaced data files during commit retries [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #3811:
URL: https://github.com/apache/iceberg-python/pull/3811#discussion_r3823683070


##########
pyiceberg/table/update/snapshot.py:
##########


Review Comment:
   i asked codex, i believe this might be the issue youre referring to: https://github.com/apache/iceberg-python/issues/3814



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Validate replaced data files during commit retries [iceberg-python]

Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu merged PR #3811:
URL: https://github.com/apache/iceberg-python/pull/3811


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org