You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2026/02/16 15:32:09 UTC

[PR] Core: Add delete deduplication optimization for equality deletes in CDC scenarios [iceberg]

wangyum opened a new pull request, #15337:
URL: https://github.com/apache/iceberg/pull/15337

   ## Description
   
   Adds delete deduplication optimization to `BaseEqualityDeltaWriter` to reduce excessive equality delete files in Flink CDC and other high-frequency update scenarios.
   
   ### Problem
   In Flink CDC workloads, the same row keys are often deleted multiple times within a single checkpoint:
   - Each UPDATE generates DELETE + INSERT in CDC stream
   - Flink processes records one-by-one without batch deduplication opportunity
   - Results in 5-10x more delete records than necessary
   - Creates excessive small delete files, causing NameNode RPC pressure
   
   ### Solution
   Add checkpoint-scoped deduplication caches to skip redundant delete operations:
   - `pendingDeleteKeys`: Deduplicates `deleteKey()` calls (key-only deletes)
   - `pendingDeleteRows`: Deduplicates `delete()` calls (full-row deletes)
   - Caches are cleared when writer closes (memory bounded by checkpoint size)
   - Check-before-copy optimization minimizes object allocations
   
   ### Performance Impact
   
   **Benchmark results:**
   
   | Scenario | Delete Ops | Without Dedup | With Dedup | Reduction |
   |----------|-----------|---------------|------------|-----------|
   | High-frequency CDC (1K orders × 10 updates) | 10,000 | 10,000 records | 1,000 records | **90%** ✅ |
   | IoT telemetry (10K devices × 6 updates) | 60,000 | 60,000 records | 10,000 records | **83%** ✅ |
   | Production pattern (1K keys, bursty) | ~7,500 | 7,500 records | 1,000 records | **87%** ✅ |
   
   **Memory overhead:**
   - ~10-15 bytes per unique delete key
   - 1,000 keys ≈ 13 KB
   - 10,000 keys ≈ 130 KB
   - Negligible for typical Flink memory (1-4 GB per task)
   
   ### Testing
   - Added `TestDeleteDeduplication.java` with 3 test cases
   - Verifies deduplication correctness
   - Ensures no false positives for unique deletes
   - All existing tests pass
   
   Closes #15336 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Add delete deduplication optimization for equality deletes in CDC scenarios [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15337:
URL: https://github.com/apache/iceberg/pull/15337#issuecomment-4086576472

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Add delete deduplication optimization for equality deletes in CDC scenarios [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15337:
URL: https://github.com/apache/iceberg/pull/15337#issuecomment-3911030888

   If there are multiple updates for the same key then the CDC stream should contain multiple DELETE + INSERT pairs.
   It should create a single equality delete, and every consecutive delete should be converted to a position delete instead which removes the newly inserted row of the previous update.
   The decision happens here:
   https://github.com/apache/iceberg/blob/ebaafdeb35ed892e95f1d9b4666a18d393c95876/core/src/main/java/org/apache/iceberg/io/BaseTaskWriter.java#L234-L241


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Add delete deduplication optimization for equality deletes in CDC scenarios [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15337:
URL: https://github.com/apache/iceberg/pull/15337#issuecomment-4130717076

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Add delete deduplication optimization for equality deletes in CDC scenarios [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #15337: Core: Add delete deduplication optimization for equality deletes in CDC scenarios
URL: https://github.com/apache/iceberg/pull/15337


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org