You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "zhaoyudi-creator (via GitHub)" <gi...@apache.org> on 2026/09/11 11:39:50 UTC

[I] [SUPPORT] Bucket Index + Flink streaming write: duplicate fileId in the same bucket after failover/savepoint restore (first write to an empty partition) [hudi]

zhaoyudi-creator opened a new issue, #19907:
URL: https://github.com/apache/hudi/issues/19907

### Describe the problem you faced

With **Bucket Index + Flink streaming write**, when a partition is written for the **first time** (the bucket has no committed base file yet), if the write task undergoes failover / savepoint restore before its corresponding instant is committed, a later index bootstrap throws:

```
Duplicate fileId <fileId> from bucket <bucketNo> of partition <partition> found
during the BucketStreamWriteFunction index bootstrap.
```

### To Reproduce

Verified online timeline:
1. **16:37** — Job restarts and recovers from state; TM re-sends the bootstrap event for the inflight instant `20260605161125980`.
2. **~16:37** — Write side: first record of the new partition → `bootstrapIndexIfNeed`; `20260605161125980` is still inflight → committed view is empty → allocates new **fileId-B**.
3. **16:40:14** — JM commits `20260605161125980` via the **recommit route** (`restoreEvents` / bootstrap quorum → `recommitInstant` → `commitInstant`), logs `Recommit instant 20260605161125980`; **fileId-A** lands.
4. **16:40:18** — JM opens the new instant `20260605164017942` (`startInstant`).
5. **16:40:19** — A second recommit trigger enters `recommitInstant`; since `20260605161125980` is already completed it falls into the else-branch `reset` (harmless cleanup, confirming the commit went through the recommit route).
6. **Later** — fileId-B is committed under `20260605164017942`; the bucket now holds fileId-A + fileId-B, and the next bootstrap throws `Duplicate fileId`.

### Expected behavior

After failover / savepoint restore, the write-side index bootstrap should see the bucket's still-pending fileId and reuse the same fileId, rather than creating a second file group.

### Environment Description

* Hudi version:
* Spark version:
* Flink version:
* Hive version:
* Hadoop version:
* Storage (HDFS/S3/GCS..):
* Running on Docker? (yes/no):

### Additional context

## Root cause — two execution lines that do not wait for each other:

- **Write-task side:** after restore, `bootstrapIndexIfNeed()` loads the bucket→fileId map from `getLatestFileSlices()` (**committed view only**). The old instant is still inflight, so fileId-A is invisible; `defineRecordLocation()` then allocates a **new fileId-B** (random UUID suffix) for the same bucket.
- **Coordinator side:** it commits that old inflight instant **through the recommit route** (`recommitInstant`), **asynchronously and after** the write side has already allocated fileId-B. fileId-B is later committed under the new instant.
- **Result:** the bucket ends up with two file groups (fileId-A + fileId-B), and the next bootstrap hits the `Duplicate fileId` check.

## Proposed fix & open questions

**Proposed fix:** In `bootstrapIndexIfNeed`, extend the source of the bucket→fileId map from "committed view only" to "committed ∪ fileIds written by still-pending (inflight) instants". This way the still-inflight fileId-A of a first-written partition becomes visible to bootstrap and is reused, instead of minting a new fileId-B. The approach is **non-blocking** (it does not intercept the consume path) and holds for both the `attemptId ≤ 0` and `attemptId > 0` restore branches.

**Downside:** It needs an API with proper **filtering semantics** to fetch the file slices of pending instants — the unfiltered `getLatestFileSlicesIncludingInflight` cannot be used directly, otherwise orphan files from instants that are about to be rolled back would be pulled in as valid fileIds. If no such clean API exists, the fallback is to have the coordinator broadcast the pending fileIds it already holds to the write tasks for warm-up, which is a larger change.

**We'd like the community to confirm:**
1. Is this race **actually real** (rather than a false problem already covered by some other mechanism)?
2. Is there a **better approach** (e.g. an existing filtering API, or a more suitable layer to fix it at)?

### Stacktrace

```shell

```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Bucket Index + Flink streaming write: duplicate fileId in the same bucket after failover/savepoint restore (first write to an empty partition) [hudi]

Posted by "zhaoyudi-creator (via GitHub)" <gi...@apache.org>.

zhaoyudi-creator commented on issue #19907:
URL: https://github.com/apache/hudi/issues/19907#issuecomment-5660570622

   ## Timeline
   <img width="1108" height="108" alt="Image" src="https://github.com/user-attachments/assets/a82030b3-bcae-4ea6-9486-554def505449" />
   ## Duplicate fileId
   <img width="620" height="78" alt="Image" src="https://github.com/user-attachments/assets/996afb1a-aacd-4f10-9d5c-51f7fdc1b6da" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Bucket Index + Flink streaming write: duplicate fileId in the same bucket after failover/savepoint restore (first write to an empty partition) [hudi]

Posted by "zhaoyudi-creator (via GitHub)" <gi...@apache.org>.

zhaoyudi-creator commented on issue #19907:
URL: https://github.com/apache/hudi/issues/19907#issuecomment-5660555841

   > Fixes that might be related:
   > 
   > [#5185](https://github.com/apache/hudi/pull/5185) [#5763](https://github.com/apache/hudi/pull/5763) [#7433](https://github.com/apache/hudi/pull/7433) [#8263](https://github.com/apache/hudi/pull/8263)
   
   Thanks for the reply. I've reproduced this on **1.1** and can confirm the race still exists — it doesn't appear to be covered by any existing mechanism, so this looks like a genuine bug rather than a known/handled case.
   
   ## Reproduction Steps
   
   1. Perform the **first write to an empty partition**, and stop the job after the checkpoint succeeds but before the instant finishes committing (this creates an inflight instant that needs recommit).
   2. Restart the job. The **JM-side recommit completes *after* the TM-side `bootstrapIndexIfNeed`**, so the write task mints a brand-new fileId while the committed view is still empty.
   
   ## Key Logs
   
   ```
   2026-09-14 14:47:57.461 INFO [Sink: bucket_write(t:p) (2/2)#0] BucketStreamWriteFunction
       - Loading Hoodie Table test_bucket_field_dep, with path .../test_bucket_field_dep/2026-09-13   # TM bootstrapIndexIfNeed starts
   2026-09-14 14:47:57.463 INFO [meta-event-handle] StreamWriteOperatorCoordinator
       - Recommit instant 20260914144335797                                                            # JM starts recommitting the old instant
   2026-09-14 14:49:58.131 INFO [meta-event-handle] BaseHoodieWriteClient
       - Committed 20260914144335797                                                                    # old instant (fileId-A) committed, ~2 min AFTER bootstrap
   2026-09-14 14:50:01.584 INFO [meta-event-handle] BaseHoodieWriteClient
       - Committed 20260914144958570                                                                    # new instant (fileId-B) committed
   ```
   
   Meanwhile, the following two log lines **never appear**:
   
   ```
   Should load this partition bucket %s with fileId %s
   Adding fileId %s to the bucket %s of partition
   ```
   
   These are only printed when `bootstrapIndexIfNeed` reads a fileId for the bucket from the committed view. Their absence is direct proof that **the committed view was empty at bootstrap time and fileId-A was invisible**, so the write task minted fileId-B in `defineRecordLocation`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Bucket Index + Flink streaming write: duplicate fileId in the same bucket after failover/savepoint restore (first write to an empty partition) [hudi]

Posted by "zhaoyudi-creator (via GitHub)" <gi...@apache.org>.

zhaoyudi-creator commented on issue #19907:
URL: https://github.com/apache/hudi/issues/19907#issuecomment-5675000052

   https://github.com/apache/hudi/pull/19951，I've put up a PR with a proposed fix for this。Would appreciate a maintainer's read on whether this direction is acceptable before I invest further — the PR description also notes a smaller "deterministic fileId" alternative and the
   trade-offs. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Bucket Index + Flink streaming write: duplicate fileId in the same bucket after failover/savepoint restore (first write to an empty partition) [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #19907:
URL: https://github.com/apache/hudi/issues/19907#issuecomment-5642806769

   can you supplement the "Environment Description", we have resolved this issue since 0.15.x.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Bucket Index + Flink streaming write: duplicate fileId in the same bucket after failover/savepoint restore (first write to an empty partition) [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #19907:
URL: https://github.com/apache/hudi/issues/19907#issuecomment-5673908199

   yeah, there is possibility since the index-mapping loading is async and build can start before the recommitting of last failed instant. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Bucket Index + Flink streaming write: duplicate fileId in the same bucket after failover/savepoint restore (first write to an empty partition) [hudi]

Posted by "zhaoyudi-creator (via GitHub)" <gi...@apache.org>.

zhaoyudi-creator commented on issue #19907:
URL: https://github.com/apache/hudi/issues/19907#issuecomment-5643478099

   > can you supplement the "Environment Description", we have resolved this issue since 0.15.x.
   
   Thanks for your reply. Here is the environment description: Hudi 0.13.1, Flink 1.16. If this issue has been fixed since 0.15.x, could you please provide the corresponding commit?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Bucket Index + Flink streaming write: duplicate fileId in the same bucket after failover/savepoint restore (first write to an empty partition) [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #19907:
URL: https://github.com/apache/hudi/issues/19907#issuecomment-5658956113

   Fixes that might be related:
   
   https://github.com/apache/hudi/pull/5185
   https://github.com/apache/hudi/pull/5763
   https://github.com/apache/hudi/pull/7433
   https://github.com/apache/hudi/pull/8263


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [SUPPORT] Bucket Index + Flink streaming write: duplicate fileId in the same bucket after failover/savepoint restore (first write to an empty partition) [hudi]

Posted by "zhaoyudi-creator (via GitHub)" <gi...@apache.org>.

zhaoyudi-creator commented on issue #19907:
URL: https://github.com/apache/hudi/issues/19907#issuecomment-5660534780

   > Fixes that might be related:
   > 
   > [#5185](https://github.com/apache/hudi/pull/5185) [#5763](https://github.com/apache/hudi/pull/5763) [#7433](https://github.com/apache/hudi/pull/7433) [#8263](https://github.com/apache/hudi/pull/8263)
   
   Thanks for the reply. I've reproduced this on **1.1** and can confirm the race still exists — it doesn't appear to be covered by any existing mechanism, so this looks like a genuine bug rather than a known/handled case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org