You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by xu...@apache.org on 2026/01/09 18:09:25 UTC
(hudi) branch asf-site updated: docs(blog): flink async instant generation blog (#17818)
This is an automated email from the ASF dual-hosted git repository.
xushiyan pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 7b9eeca1b132 docs(blog): flink async instant generation blog (#17818)
7b9eeca1b132 is described below
commit 7b9eeca1b13219f84cb3847495644379458f340d
Author: Shiyan Xu <27...@users.noreply.github.com>
AuthorDate: Fri Jan 9 12:09:14 2026 -0600
docs(blog): flink async instant generation blog (#17818)
---
...09-hudi-11-deep-dive-flink-async-instant-gen.md | 147 +++++++++++++++++++++
.../async-instant-gen.png | Bin 0 -> 149583 bytes
.../blocking-instant-gen.png | Bin 0 -> 150864 bytes
.../file-slicing-completion-time.png | Bin 0 -> 34356 bytes
.../file-slicing-instant-time.png | Bin 0 -> 33333 bytes
.../lsm-timeline.png | Bin 0 -> 100086 bytes
.../timeline.png | Bin 0 -> 283012 bytes
.../truetime.png | Bin 0 -> 39169 bytes
.../writemeta-failover.png | Bin 0 -> 161470 bytes
9 files changed, 147 insertions(+)
diff --git a/website/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen.md b/website/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen.md
new file mode 100644
index 000000000000..690063f66c42
--- /dev/null
+++ b/website/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen.md
@@ -0,0 +1,147 @@
+---
+title: "Apache Hudi 1.1 Deep Dive: Async Instant Time Generation for Flink Writers"
+excerpt: "Explore how Hudi 1.1 introduces asynchronous instant generation for Flink writers to eliminate throughput fluctuations in streaming ingestion."
+authors: [shuo-cheng]
+category: blog
+image: /assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/async-instant-gen.png
+tags:
+ - hudi
+ - flink
+ - streaming
+---
+
+---
+
+_This blog was translated from the [original blog in Chinese](https://mp.weixin.qq.com/s/3r06DFdkaiGkF1_7NiovZw)._
+
+---
+
+## Background
+
+Before the Hudi 1.1 release, in order to guarantee the exactly-once semantics of the Hudi Flink sink, a new instant could only be generated after the previous instant was successfully committed to Hudi. During this period, Flink writers had to block and wait. Starting from Hudi 1.1, we introduce a new asynchronous instant generation mechanism for Flink writers. This approach allows writers to request the next instant even before the previous one has been committed successfully. At the sa [...]
+
+## Instant Time
+
+Timeline is a core component of Hudi's architecture. It serves as the single source of truth for the table's state, recording all operations performed on a table. Each operation is identified by a commit with a monotonically increasing instant time, which indicates the start time of each transaction.
+
+
+
+Hudi provides the following capabilities based on instant time:
+
+- **More efficient write rollbacks**: Each Hudi commit corresponds to an instant time. The instant timestamp can be used to quickly locate files affected by failed writes.
+- **File name-based file slicing**: Since instant time is encoded into file names, Hudi can efficiently perform file slicing across different versions of files within a table.
+- **Incremental queries**: Each row in a Hudi table carries a `_hoodie_commit_time` metadata field. This allows incremental queries at any point in the timeline, even when full compaction or cleaning services are running asynchronously.
+
+## Completion Time
+
+### File Slicing Based on Instant Time
+
+Before release 1.0, Hudi organized data files in units called `FileGroup`. Each file group contains multiple `FileSlice`s. Each file slice contains one base file and multiple log files. Every compaction on a file group generates a new file slice. The timestamp in the base file name corresponds to the instant time of the compaction operation that wrote the file. The timestamp in the log file name is the same as the base instant time of the current file slice. Data files with the same inst [...]
+
+
+
+In concurrent write scenarios, the instant time naming convention of log files introduces certain limitations: as asynchronous compaction progresses, the base instant time can change. To ensure that writers can correctly determine the base instant time, the ordering between write commits and compaction scheduling must be enforced—meaning that compaction can only be scheduled when there are no ongoing write operations on the table. Otherwise, a log file might be written with an incorrect [...]
+
+### File Slicing Based on Completion Time
+
+To address these issues, starting from version 1.0, Hudi introduced a new file slicing model based on a time interval defined by requested time and completion time. In release 1.x, each commit has two important time concepts: requested time and completion time. All generated timestamps are globally monotonically increasing. The timestamp in the log file name is no longer the base instant time, but rather the requested instant time of the write operation. During the file slicing process, [...]
+
+> _A log file belongs to the file slice with the maximum base requested time that is less than or equal to the log file's completion time._ [5]
+
+
+
+The new file slicing mechanism is more flexible, allowing compaction scheduling to be completely decoupled from the writer's ingestion process. Based on this mechanism, Hudi has also implemented powerful non-blocking concurrency control. For details, refer to RFC-66 [5].
+
+### LSM Timeline
+
+The new file slicing mechanism requires efficient querying of completion time based on instant time. Starting from version 1.x, Hudi re-implemented the archived timeline. The new archived timeline organizes its data files in an LSM-tree structure, enabling fast range-filtering queries based on instant time and supporting efficient data skipping. For more details, refer to [6].
+
+
+
+## TrueTime
+
+A critical premise of Hudi's timeline is that the timestamp generated for instants must be globally monotonically increasing without any conflicts. However, maintaining time monotonicity in distributed transactions has long been a thorny problem due to:
+
+- **Clock skew**: Physical clocks drift apart across machines.
+- **Network latency**: Communication delays between data centers.
+- **Concurrent transactions**: Difficulty in determining event ordering across regions.
+
+To solve this problem, Google Spanner's TrueTime [1] uses a globally synchronized clock with bounded uncertainty, allowing it to assign timestamps monotonically without conflicts. Similarly, Hudi introduced the TrueTime API [2] starting from the 1.0 release. There are generally two approaches to realize TrueTime semantics:
+
+- A single shared time generator process or service, like Google Spanner's time service.
+- Each process generates its own time and waits until time >= maximum expected clock drift across all processes, coordinated within a distributed lock.
+
+
+
+Hudi's TrueTime API adopts the second approach: using a distributed lock to ensure only one process generates time at any given moment. The waiting mechanism ensures sufficient time passes so that the instant time is monotonically increasing globally.
+
+## Blocking Instant Time Generation for Flink Writers
+
+For Flink streaming ingestion, each incremental transaction write can be roughly divided into the following stages:
+
+1. Writers write records into the in-memory buffer.
+2. When the buffer is full or a checkpoint is triggered, writers send a request to the coordinator for an instant time.
+3. Writers create data files based on the instant time and flush the data to storage.
+4. The coordinator commits the flushed files and metadata after receiving the ACK event for the successful checkpoint.
+
+
+
+From Hudi's file slicing mechanism, we can see that the instant time is required before writers flush records into storage. Before 1.1, although the committing operation was performed asynchronously in the coordinator, the writer's request for a new instant would be blocked, because only after the previous instant is successfully committed can the coordinator create a new instant. Let's say a batch of records finished flushing with checkpoint `ckp_1` in the writer at `T1`, and `ckp_1` co [...]
+
+## Async Instant Time Generation
+
+To address throughput fluctuation caused by blocking instant generation, Hudi 1.1 introduces asynchronous instant generation for Flink writers. Using the aforementioned example, "async" means that before the previous instant is successfully committed to the timeline (during the time range `[T1, T2]`), the coordinator can create a new instant and return it to the writer for flushing a new batch of data. Thus, writers are no longer blocked during checkpoints, and the ingestion throughput w [...]
+
+
+
+**Overall write workflow:**
+
+- **Writer**: Requests instant time from the coordinator before flushing data:
+ - Data flushing may be triggered either by a checkpoint or by the buffer being full, so the request carries the last completed checkpoint ID, rather than the current checkpoint ID.
+ - For a request made at the initial startup of a task, if it's recovered from state, the checkpoint ID is fetched from the state; otherwise, it's set to -1.
+- **Coordinator**: Responsible for instant generation. Each checkpoint ID corresponds to an instant. The coordinator maintains the mapping in the WriteMeta buffer: `CheckpointID → { Instant → { Writer TaskID → Writer MetaEvent }}`, where `{Writer TaskID → Writer MetaEvent}` is the mapping between each writer's parallel task ID and the file metadata written during the current checkpoint interval. Upon receiving an instant request from a writer:
+ - If an instant corresponding to the checkpoint ID already exists in the WriteMeta buffer, the cached instant is returned directly.
+ - Otherwise, a new instant is generated, added to the WriteMeta buffer, and then returned to the writer.
+- **Writer**: Completes data writing, generates data files, and sends the file metadata to the coordinator.
+- **Coordinator**: Upon receiving the file metadata from a writer, updates the WriteMeta buffer:
+ - If there is no existing metadata for the current writer task in the buffer, the metadata is written directly into the cache.
+ - If metadata for the current writer task already exists (i.e., the writer has flushed multiple times), the metadata is merged and then the cache is updated.
+- **Coordinator**: When the ACK event for a checkpoint (ID = n) is received, it starts serially and orderly committing the instants recorded in the WriteMeta buffer. The commit scope includes all instants corresponding to checkpoint IDs < n. Once committed successfully, the instants are removed from the buffer.
+ - Since Flink's checkpoint ACK mechanism does not guarantee that the coordinator will receive an ACK for every checkpoint, the commit logic follows Flink's Checkpoint Subsume Contract [4]: if the ACK for checkpoint `ckp_i` is not received, its written metadata is subsumed into the pending metadata for the next checkpoint `ckp_i+1`, and will be committed once the ACK for `ckp_i+1` is received.
+
+### WriteMeta Failover
+
+Considering that the coordinator cannot guarantee receiving every checkpoint ACK event, the in-memory WriteMeta buffer needs to be persisted to Flink state to prevent data loss after a task failover.
+
+When designing the snapshot for the WriteMeta buffer, the following points need to be considered:
+
+- WriteMeta is initially stored in the writer's buffer. Only when a checkpoint is triggered or an eager flush occurs does the writer send the current WriteMeta to the coordinator.
+- During checkpointing, the coordinator takes a snapshot first, followed by the writers' snapshot. At this point, the coordinator's cache does not yet include the WriteMeta for the current checkpoint interval.
+
+
+
+Based on the above considerations, the WriteMeta snapshot process for checkpoint i is as follows:
+
+1. The coordinator first persists the WriteMeta for all checkpoint IDs < i to Flink state.
+2. The writer sends the WriteMeta generated during checkpoint i to the coordinator, awaiting commit.
+3. The writer cleans up the historical WriteMeta in state and persists the WriteMeta for checkpoint i to Flink state.
+
+This approach of persisting the WriteMeta buffer ensures that metadata is not lost, while also preventing state bloat in the WriteMeta state.
+
+## Conclusion
+
+The async instant time generation mechanism introduced in Hudi 1.1 is an important feature for improving the stability of Flink streaming ingestion. By eliminating the blocking dependency on the completion of the previous instant's commit, this mechanism solves the throughput fluctuation and backpressure problems under large-scale workloads. At the same time, it still maintains strong transactional guarantees and seamless integration with Flink's checkpoint semantics. This enhancement is [...]
+
+## References
+
+[1] [https://research.google/pubs/spanner-truetime-and-the-cap-theorem/](https://research.google/pubs/spanner-truetime-and-the-cap-theorem/)
+
+[2] [https://hudi.apache.org/docs/next/timeline/#timeline-components](https://hudi.apache.org/docs/next/timeline/#timeline-components)
+
+[3] [https://github.com/apache/hudi/blob/master/rfc/rfc-66/rfc-66.md](https://github.com/apache/hudi/blob/master/rfc/rfc-66/rfc-66.md)
+
+[4] [https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/state/CheckpointListener.java](https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/state/CheckpointListener.java)
+
+[5] [https://github.com/apache/hudi/blob/master/rfc/rfc-66/rfc-66.md](https://github.com/apache/hudi/blob/master/rfc/rfc-66/rfc-66.md)
+
+[6] [https://hudi.apache.org/docs/next/timeline/#lsm-timeline-history](https://hudi.apache.org/docs/next/timeline/#lsm-timeline-history)
diff --git a/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/async-instant-gen.png b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/async-instant-gen.png
new file mode 100644
index 000000000000..94959c3791bc
Binary files /dev/null and b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/async-instant-gen.png differ
diff --git a/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/blocking-instant-gen.png b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/blocking-instant-gen.png
new file mode 100644
index 000000000000..0c0afd39a787
Binary files /dev/null and b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/blocking-instant-gen.png differ
diff --git a/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/file-slicing-completion-time.png b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/file-slicing-completion-time.png
new file mode 100644
index 000000000000..0af0fd092d94
Binary files /dev/null and b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/file-slicing-completion-time.png differ
diff --git a/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/file-slicing-instant-time.png b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/file-slicing-instant-time.png
new file mode 100644
index 000000000000..961ae919c115
Binary files /dev/null and b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/file-slicing-instant-time.png differ
diff --git a/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/lsm-timeline.png b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/lsm-timeline.png
new file mode 100644
index 000000000000..b4f282e1bf54
Binary files /dev/null and b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/lsm-timeline.png differ
diff --git a/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/timeline.png b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/timeline.png
new file mode 100644
index 000000000000..c47f8ee252a2
Binary files /dev/null and b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/timeline.png differ
diff --git a/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/truetime.png b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/truetime.png
new file mode 100644
index 000000000000..f828b6b02171
Binary files /dev/null and b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/truetime.png differ
diff --git a/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/writemeta-failover.png b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/writemeta-failover.png
new file mode 100644
index 000000000000..97e5a0614a25
Binary files /dev/null and b/website/static/assets/images/blog/2026-01-09-hudi-11-deep-dive-flink-async-instant-gen/writemeta-failover.png differ