You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/06/25 02:28:24 UTC

Re: [I] HoodieAppendhandle may result in duplicate key for hbase index [hudi]

nsivabalan commented on issue #14957:
URL: https://github.com/apache/hudi/issues/14957#issuecomment-4795291678

   Triage: this should be fixed on `master` for table version 8+ (current default is v9).
   
   The earlier PR #4444 (HUDI-1517) linked here was reverted in commit `3d9801a9` (Jul 2023). The proper fix was later introduced via **HUDI-9236** in commit [`14e40699`](https://github.com/apache/hudi/commit/14e406992752b7f7dd1d4323f70df9fa927bd637) (Mar 2025, PR #13007: "Handle markers for log files in table version 6").
   
   In current master:
   - `HoodieWriteHandle.getLogCreationCallback()` creates a marker (`IOType.CREATE` for table v8+) every time a log file is created, including the spilled-over file group and retried task attempts.
   - `HoodieTable.reconcileAgainstMarkers()` → `getInvalidDataPaths()` uses `createdAndMergedDataPaths`, which now includes CREATE markers for log files. Any log file with a marker but not in `writeStats` (i.e. the orphan from a failed task attempt described in the repro) is deleted before commit, so the duplicate-key scenario across two file groups can no longer occur.
   
   For table version 6, the approach is different — `MarkerBasedCommitMetadataResolver` adds the orphan log files into the commit metadata (so MDT stays consistent) rather than deleting them. If the original bug scenario is still reproducible specifically on table v6, please reopen with that detail; otherwise this can be closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org