Re: [I] Potential duplicates in Create Handle after partition update [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #15852:
URL: https://github.com/apache/hudi/issues/15852#issuecomment-4795489849

   Triage update: this looks resolved on current `master`.
   
   The originally linked PR #8228 (which would have deduplicated inside `HoodieCreateHandle`) was closed without merging. The duplicate-on-partition-update concern was instead addressed upstream of the write handle, in the global-index tagging path:
   
   - HUDI-5968 / commit `cabcb2bf2cd` — `Fix global index duplicate and handle custom payload when update partition` (May 2023)
   - HUDI-9602 / commit `4ebc2545245e` — `Use BufferedRecordMerger for dedup and global index path` (Aug 2025)
   
   In current master, `HoodieIndexUtils.mergeForPartitionUpdatesAndDeletionsIfNeeded` applies `distinctWithKey(HoodieRecord::getRecordKey, ...)` on the updating records (see `HoodieIndexUtils.java:484`) and then merges with existing records, so the create-handle ultimately receives at most one record per key for the new partition. If anyone is still seeing duplicates in this scenario on a recent release, please share a repro and I can reopen — otherwise this can probably be closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org