You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/07/23 22:12:57 UTC

[PR] fix(timeline-service): fail marker creation requests when marker flush fails [hudi]

yihua opened a new pull request, #19368:
URL: https://github.com/apache/hudi/pull/19368

### Describe the issue this Pull Request addresses

`MarkerDirState.flushMarkersToFile` in the timeline server writes the `MARKERS<i>` file and closes the output stream with `closeQuietly` in a `finally` block. On storage that uploads the file content at `close()` time (for example S3A, which performs the object PUT inside `close()`), a failed flush is silently swallowed, and `processMarkerCreationRequests` then completes the pending marker creation futures as successful. The writer receives success and writes data files that have no durable marker. Marker-based rollback of a failed write only sees files with persisted markers, so it misses such files, and retrying the write under the same instant (for example, a clustering retry) can leave duplicate file groups behind that are visible to listing-based readers.

### Summary and Changelog

Marker creation requests to the timeline server now fail when the markers cannot be durably persisted, instead of being falsely acknowledged as successful.

- `flushMarkersToFile` closes the writer within the try scope (try-with-resources), so a failure to persist the markers, including at `close()` time, propagates as `HoodieIOException` instead of being swallowed.
- On a failure to persist, `processMarkerCreationRequests` removes the markers buffered by the failed batch from the in-memory state and completes the pending marker creation futures exceptionally. The marker creation requests fail on the writer side (the timeline server returns an error response), and a retried request can recreate the markers.
- The marker file index in use is released in a `finally` block, so a failure does not leak the file index.
- Adds `TestMarkerDirState` that simulates a `close()` failure and asserts that the marker creation request fails, the in-memory marker state is rolled back, the file index is released, and a retried request for the same marker succeeds once the storage recovers.

### Impact

No public API change. Writers using timeline-server-based markers now get a failed marker creation request (instead of a false success) when the marker file cannot be persisted, so data files are not written without durable markers and marker-based rollback remains complete.

### Risk Level

low. New unit tests cover the success, failure, and retry-after-failure paths of marker creation request processing.

### Documentation Update

none

### Contributor's checklist

- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org