You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "jordepic (via GitHub)" <gi...@apache.org> on 2026/04/14 15:23:20 UTC
[PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
jordepic opened a new pull request, #15970:
URL: https://github.com/apache/iceberg/pull/15970
Equality delete files require an expensive hash-join at read time, degrading scan performance proportionally to mutation volume. This adds a Spark stored procedure and action that converts equality deletes to deletion vectors (Roaring bitmaps in Puffin files), eliminating the join at read time.
Key design decisions:
- Per-partition processing with configurable parallelism to bound memory and enable progress isolation
- Files read via FormatModelRegistry with field-ID matching, supporting Parquet (vectorized), ORC (vectorized), and Avro
- Equality-schema groups joined independently within each partition, then positions unioned, deduplicated, and written as DVs
- Join strategy configurable (auto/broadcast/hash/sort-merge)
- Filter expression support via inclusive partition projection
- Existing DVs merged during write
- DVs written in the executors themselves to avoid pressure on driver
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
Posted by "tomerr90 (via GitHub)" <gi...@apache.org>.
tomerr90 commented on PR #15970:
URL: https://github.com/apache/iceberg/pull/15970#issuecomment-6060936340
1. The code is no longer available it seems (I guess you deleted your fork?)
2. Thank you! wasnt aware of this project, will take a look
3. Interesting, what makes you want to create something like that when thats exactly what the Iceberg community is going to implement for Flink as part of V4 (AFAIK)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic commented on PR #15970:
URL: https://github.com/apache/iceberg/pull/15970#issuecomment-6060973973
1) Hmmm, I'll see if I can find it but may be out of luck
2) nice!
3) I did it before it was announced!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
Posted by "tomerr90 (via GitHub)" <gi...@apache.org>.
tomerr90 commented on PR #15970:
URL: https://github.com/apache/iceberg/pull/15970#issuecomment-6058645488
Hey @jordepic !
Do you still have the code for this? Wondering why you closed this?
It is very useful for streaming datalakes today that have to run rewrite_data_files pretty frequently to avoid equality deletes pile up, instead this can be run which should be much lighter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic commented on PR #15970:
URL: https://github.com/apache/iceberg/pull/15970#issuecomment-4246417534
Hi @singhpk234 - I just ammended the commit with a markdown file!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic commented on PR #15970:
URL: https://github.com/apache/iceberg/pull/15970#issuecomment-6059753893
Gotcha. I don't think this one ever had much interest associated with it :(.
1) You could port the code yourself and run it in a spark jar
2) Apache amoro allows you to do equality deletes to positional deletes
3) I created an OSS project called icestream which does it too, but it's not very mature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic closed pull request #15970: Spark: Add convert_equality_deletes procedure and action
URL: https://github.com/apache/iceberg/pull/15970
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on PR #15970:
URL: https://github.com/apache/iceberg/pull/15970#issuecomment-4246256372
Thank you for the contribution @jordepic ! Do you have a considated design doc explaining the design choices and tradeoff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
Posted by "jordepic (via GitHub)" <gi...@apache.org>.
jordepic commented on PR #15970:
URL: https://github.com/apache/iceberg/pull/15970#issuecomment-6059656132
Are you able to use the new flink maintainance procedure which converts equality deletes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: Add convert_equality_deletes procedure and action [iceberg]
Posted by "tomerr90 (via GitHub)" <gi...@apache.org>.
tomerr90 commented on PR #15970:
URL: https://github.com/apache/iceberg/pull/15970#issuecomment-6059710434
It has problematic limitations like having to run the compaction job as part of the job that also does the writing, so no.
I think a more common use case (maybe Im completely wrong) is to have writers separate from whats running the compaction (usually spark)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org