You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "tomlarkworthy (via GitHub)" <gi...@apache.org> on 2025/11/20 16:11:55 UTC
[PR] Equality delete tests [iceberg-python]
tomlarkworthy opened a new pull request, #2773:
URL: https://github.com/apache/iceberg-python/pull/2773
I've successfully created a proof-of-concept demonstrating that PyIceberg already supports writing equality delete files via transactions, even though the read path is not yet implemented.
What I Discovered
1. No tests use actual equality_ids values - All existing tests either set it to [] or None
2. The write infrastructure is complete and working - All necessary components exist:
- DataFileContent.EQUALITY_DELETES enum
- equality_ids field in DataFile
- Snapshot tracking for equality deletes
- Manifest serialization
3. The key is using the transaction API directly:
with table.transaction() as txn:
update_snapshot = txn.update_snapshot()
with update_snapshot.fast_append() as append_files:
append_files.append_data_file(delete_file) # Works for delete files!
Files Created
1. test_equality_delete_poc.py - Detailed standalone test with verbose output
2. test_add_equality_delete.py - Clean pytest suite with 2 passing tests:
- Single equality delete file
- Multiple delete files with different equality_ids
3. example_add_equality_delete.py - Complete working examples showing:
- Basic usage (single column)
- Composite keys (multiple columns)
- Multiple delete files in one transaction
4. EQUALITY_DELETE_POC_SUMMARY.md - Comprehensive documentation
Test Results
All tests pass successfully:
test_add_equality_delete.py::test_add_equality_delete_file_via_transaction PASSED
test_add_equality_delete.py::test_add_multiple_equality_delete_files_with_different_equality_ids PASSED
====== 2 passed in 1.06s ======
Key Takeaways
- ✅ You can write equality delete files today using the transaction API
- ✅ Single column deletes: equality_ids=[1]
- ✅ Composite key deletes: equality_ids=[1, 2]
- ✅ Multiple delete files can be added in one transaction
- ✅ Metadata tracking works correctly (snapshot summaries, manifests)
- ❌ Reading is blocked - raises ValueError when scanning tables with equality deletes
The write path is production-ready. Users who generate equality delete files externally can add them to PyIceberg tables now, though they'll need other tools (like Spark) to read those tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Equality delete tests [iceberg-python]
Posted by "tomlarkworthy (via GitHub)" <gi...@apache.org>.
tomlarkworthy commented on PR #2773:
URL: https://github.com/apache/iceberg-python/pull/2773#issuecomment-3558902101
this was a mistake!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Equality delete tests [iceberg-python]
Posted by "tomlarkworthy (via GitHub)" <gi...@apache.org>.
tomlarkworthy closed pull request #2773: Equality delete tests
URL: https://github.com/apache/iceberg-python/pull/2773
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org