You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kinolaev (via GitHub)" <gi...@apache.org> on 2026/06/24 17:22:30 UTC
[PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
kinolaev opened a new pull request, #16956:
URL: https://github.com/apache/iceberg/pull/16956
Equality delete files can be referenced by many data files. This PR allows to accumulate all equality deletes of a table snapshot in the structure `spec id -> partition -> equality field ids -> equality key -> max data sequence number` that can be reused for many data files. If passed to `DeleteFilter`, it fills the structure lazily.
I added shared equality deletes to Spark's `RowDataReader` and `BatchDataReader` by default, because the size of the shared structure is bounded by split size (multiplied by compression rate). To `GenericReader` I added a boolean parameter `shareEqDeletes` which is false by default, because the size is unbounded, and switching the parameter to true might cause OOMs. And I don't use Flink, so I hesitated to propose changing its behavior.
The non-shared `DeleteFilter` path is unchanged, so Flink and other engines are byte-for-byte identical. The shared path produces identical results (equivalence covered by tests).
Additionally, this PR changes task weight calculation algorithm - each delete file is accounted only once per bin even if it is referenced by several data files in that bin. This change can be moved into a separate PR if preferred.
In our case these changes reduced Spark memory consumption by 30-50% and, more important, prevented Spark from hanging for hours on building and evicting huge equality delete sets for each data file. For frequently updated tables (several thousands of equality delete files per day) ingested with [Debezium Iceberg Sink](https://github.com/memiiso/debezium-server-iceberg) `rewrite_data_files` execution time is reduced from several hours to several minutes.
Btw, I found that Trino uses a similar technique: [EqualityDeleteFilter](https://github.com/trinodb/trino/blob/481/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/delete/EqualityDeleteFilter.java), [DeleteManager](https://github.com/trinodb/trino/blob/481/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/delete/DeleteManager.java).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#issuecomment-5398883502
The conflict is resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#issuecomment-5084063362
The PR is not stale, it is awaiting review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#issuecomment-5971874433
Rebased, applied changes for Spark 4.2, and moved changes for other Spark versions to [a separate branch](https://github.com/kinolaev/iceberg/compare/fix-equality-delete-task-weight...kinolaev:iceberg:fix-equality-delete-task-weight-backport) for a backport PR after this one is merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#issuecomment-6127397343
@aokolnychyi, could you please take a look at this PR when you get a chance? It improves equality delete read performance by building an equality delete structure shared across all data files in a task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#issuecomment-6014992120
@szehon-ho, could you please review this PR when you have a moment? I would also appreciate it if you could ask someone else interested in optimizing equality delete planning and reading for a review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#issuecomment-4792553805
Hi @singhpk234 , I didn't try it, because I have only one big executor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#issuecomment-5523300467
@huaxingao, could you please take a look at this PR when you get a chance? This is also related to equality deletes; it significantly improves read performance and memory consumption by lazily building a shared equality delete map for all data files in a Spark task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#issuecomment-5081231917
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Build equality delete map once per task [iceberg]
Posted by "kinolaev (via GitHub)" <gi...@apache.org>.
kinolaev commented on PR #16956:
URL: https://github.com/apache/iceberg/pull/16956#issuecomment-6467974169
@danielcweeks, could you please review this PR when you have a moment? I know equality deletes are being deprecated in v4, but the default format version is still 2, so this optimization for reading equality deletes could still save a lot of people headaches and compute until we have an easy open-source CDC solution with upsert support based solely on DVs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org