You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amitpoorab (via GitHub)" <gi...@apache.org> on 2026/08/11 19:50:35 UTC
[PR] Fix/3758 partition predicate rootcause [iceberg-python]
amitpoorab opened a new pull request, #3782:
URL: https://github.com/apache/iceberg-python/pull/3782
<!--
Thanks for opening a pull request!
-->
<!-- In the case this PR will resolve an issue, please replace ${GITHUB_ISSUE_ID} below with the actual Github issue id. -->
<!-- Closes #3758 -->
# Rationale for this change
On temporal-partitioned tables, `upsert()` was silently corrupting data by leaving replaced rows in place and duplicating untouched rows. `Table.delete()` on partial rewrites had the same issue.
**Root cause:** Manifest pruning derived partition predicates from source columns + partition values, then re-applied transforms during projection. This double-transformation caused false negatives in manifest evaluation, missing the manifest holding the file being replaced.
**The fix:** Build partition-space predicates directly using partition field names (e.g., `ts_day` instead of `ts`), avoiding re-transformation. This works for ALL transform types (identity, temporal day/month/year/hour, bucket, truncate) — not just identity.
**Credits:** Commits cherry-picked from @paulcaron16k's investigation branch. They implemented the fix; I diagnosed the root cause and brought it forward based on maintainer feedback.
## Are these changes tested?
Yes. Added comprehensive regression tests:
1. **`test_upsert_partial_rewrite_of_partitioned_file`** — Tests `upsert()` on 7 transform types:
- IdentityTransform (control)
- TruncateTransform (idempotent, control)
- YearTransform, MonthTransform, DayTransform, HourTransform (temporal — regression cases)
- BucketTransform (hash-based)
## Are there any user-facing changes?
Yes. This fixes a data corruption bug (regression from v0.11.1) in `upsert()` and `delete()` operations on temporal-partitioned tables. Users with day/month/year/hour-partitioned tables will now get correct results instead of silent data loss.
<!-- In the case of user-facing changes, please add the changelog label. -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix issue #3758: manifest pruning for transformed partitions in upsert/delete [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu closed pull request #3782: Fix issue #3758: manifest pruning for transformed partitions in upsert/delete
URL: https://github.com/apache/iceberg-python/pull/3782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix/3758 partition predicate rootcause [iceberg-python]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on PR #3782:
URL: https://github.com/apache/iceberg-python/pull/3782#issuecomment-5262233130
Could you provide a more appropriate PR title?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix issue #3758: manifest pruning for transformed partitions in upsert/delete [iceberg-python]
Posted by "amitpoorab (via GitHub)" <gi...@apache.org>.
amitpoorab commented on PR #3782:
URL: https://github.com/apache/iceberg-python/pull/3782#issuecomment-5262397111
Done. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Fix issue #3758: manifest pruning for transformed partitions in upsert/delete [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #3782:
URL: https://github.com/apache/iceberg-python/pull/3782#issuecomment-5335162481
Thanks for the PR, I'm going to close it since #3758 has already been addressed
context: https://github.com/apache/iceberg-python/issues/3758#issuecomment-5331741698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org