You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "daviddallakyan2005 (via GitHub)" <gi...@apache.org> on 2026/08/16 15:56:57 UTC
[PR] Add equality-delete apply on PyArrow reads [iceberg-python]
daviddallakyan2005 opened a new pull request, #3805:
URL: https://github.com/apache/iceberg-python/pull/3805
Closes #3270
# Rationale for this change
Index equality delete files and apply them on PyArrow scans so CDC and merge-on-read tables can be read. Equality deletes still cannot be written.
This ports the DeleteFileIndex work from #3285 onto current main, skips bounds prune when an equality field has been dropped, and left anti joins matching equality delete rows after positional deletes. It supersedes #3285, which is open and conflicting. Credit to Alex Stephen for the index. Please close #3285 if this path is preferred.
When a data file has equality deletes, the scan concatenates that file’s record batches, aligns each matching equality-delete table by field id then concats, adds is_null / fill_null (and is_nan for floats) helper columns per key, left anti joins once, then yields the survivors; files with only position deletes still yield each scanner batch.
## Are these changes tested?
Yes.
`make test`
3962 passed, 3 skipped, 1570 deselected
`make lint`
All prek hooks Passed
`make test PYTEST_ARGS="-v -k equality"`
168 passed, 5367 deselected
`uv run python -m pytest tests/table/test_delete_file_index.py tests/io/test_pyarrow.py tests/catalog/test_scan_planning_models.py tests/table/test_validate.py tests/table/test_init.py -m "(unmarked or parametrize) and not integration" -v`
371 passed, 57 deselected
`make test-integration PYTEST_ARGS="-v --tb=short"`
1350 passed, 71 skipped, 3 failed, 4111 deselected
The three integration failures match main without this diff. `test_read_spark_written_puffin_dv` raises DeprecationWarning for `puffin.to_vector()`. Two DuckDB tests failed to download the iceberg extension.
## Are there any user-facing changes?
Yes. Tables that already store equality deletes can be scanned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add equality-delete apply on PyArrow reads [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3805:
URL: https://github.com/apache/iceberg-python/pull/3805#issuecomment-5723153069
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add equality-delete apply on PyArrow reads [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3805: Add equality-delete apply on PyArrow reads
URL: https://github.com/apache/iceberg-python/pull/3805
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Add equality-delete apply on PyArrow reads [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3805:
URL: https://github.com/apache/iceberg-python/pull/3805#issuecomment-5824606750
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org