You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by vi...@apache.org on 2024/12/06 13:40:24 UTC
(hudi) branch asf-site updated: [HUDI-8671][DOCS] Improve record merger docs (#12440)
This is an automated email from the ASF dual-hosted git repository.
vinoth pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new c32f1a754be [HUDI-8671][DOCS] Improve record merger docs (#12440)
c32f1a754be is described below
commit c32f1a754be60677b5d8e638b52eaae80215801a
Author: Y Ethan Guo <et...@gmail.com>
AuthorDate: Fri Dec 6 05:40:18 2024 -0800
[HUDI-8671][DOCS] Improve record merger docs (#12440)
---
website/docs/record_merger.md | 21 +++++++++++++++------
1 file changed, 15 insertions(+), 6 deletions(-)
diff --git a/website/docs/record_merger.md b/website/docs/record_merger.md
index 983b0f63da9..d98a5fc462a 100644
--- a/website/docs/record_merger.md
+++ b/website/docs/record_merger.md
@@ -8,7 +8,7 @@ toc_max_heading_level: 4
Hudi handles mutations to records and streaming data, as we briefly touched upon in [timeline ordering](/docs/next/timeline#ordering-of-actions) section.
To provide users full-fledged support for stream processing, Hudi goes all the way making the storage engine and the underlying storage format
-understand how to merge change changes to the same record key, that may arrive even in different order at different times. With the rise of mobile applications
+understand how to merge changes to the same record key, that may arrive even in different order at different times. With the rise of mobile applications
and IoT, these scenarios have become the normal than an exception. For e.g. a social networking application uploading user events several hours after they happened,
when the user connects to WiFi networks.
@@ -30,7 +30,7 @@ different merge results when switching between the modes.
### COMMIT_TIME_ORDERING
-This is the default merge mode. Here, we expect the input records to arrive in strict order such that arrival order is same as their
+Here, we expect the input records to arrive in strict order such that arrival order is same as their
delta commit order on the table. Merging simply picks the record belonging to the latest write as the merged result. In relational data mode speak,
this provides overwrite semantics aligned with serializable writes on the timeline.
@@ -43,7 +43,7 @@ that denotes the ordering of the writes in the upstream database.
### EVENT_TIME_ORDERING
-While commit time ordering provides a well-understood standard behavior, it's hardly sufficient. The commit time is unrelated to the actual
+This is the default merge mode. While commit time ordering provides a well-understood standard behavior, it's hardly sufficient. The commit time is unrelated to the actual
ordering of data that a user may care about and strict ordering of input in complex distributed systems is difficult to achieve.
With event time ordering, the merging picks the record with the highest value on a user specified _**ordering or precombine field**_ as the merged result.
@@ -52,7 +52,7 @@ With event time ordering, the merging picks the record with the highest value on
</figure>
In the example above, two microservices product change records about orders at different times, that can arrive out-of-order. As color coded,
-this can lead to application level inconsistent states in the table if simply merged in commit time order like a cancelled order being re-created or
+this can lead to application-level inconsistent states in the table if simply merged in commit time order like a cancelled order being re-created or
a paid order moved back to just created state expecting payment again. Event time ordering helps by ignoring older state changes that arrive late and
avoiding order status from "jumping back" in time. Combined with [non-blocking concurrency control](/docs/next/concurrency_control#non-blocking-concurrency-control-mode),
this provides a very powerful way for processing such data streams efficiently and correctly.
@@ -67,7 +67,7 @@ across order_info and payment_info without costly self-join on each access.
Hudi allows authoring of cross-language custom record mergers on top of a standard record merger API, that supports full and partial merges. The java APIs
are sketched below at a high-level. It simply takes older/newer records in engine native formats and produces a merged record or returns empty to skip them entirely (e.g. soft deletes).
-Record merger is configured using a `hoodie.record.merge.strategy.id` table config whose value is an uuid, that is expected to be returned by `getMergingStrategy()`
+Record merger is configured using a `hoodie.write.record.merge.strategy.id` write config whose value is an uuid, that is taken by the writer to persist in the table config, and is expected to be returned by `getMergingStrategy()`
method below. Using this mechanism, Hudi can automatically deduce the record merger to use for the table across different language/engine runtimes.
```Java
@@ -91,6 +91,15 @@ interface HoodieRecordMerger {
}
```
+### Record Merge Configs
+
+The record merge mode and optional record merge strategy ID and custom merge implementation classes can be specified using the below configs.
+
+| Config Name | Default | Description |
+| ---------------------------------------| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hoodie.write.record.merge.mode | EVENT_TIME_ORDERING | Determines the logic of merging different records with the same record key. Valid values: (1) `COMMIT_TIME_ORDERING`: use commit time to merge records, i.e., the record from later commit overwrites the earlier record with the same key. (2) `EVENT_TIME_ORDERING` (default): use event time as the ordering to merge records, i.e., the record with the larger event time overwrites the record with the smaller event time on the same key, r [...]
+| hoodie.write.record.merge.strategy.id | N/A (Optional) | ID of record merge strategy. When you specify this config, you also need to specify `hoodie.write.record.merge.custom.implementation.classes`. Hudi picks the `HoodieRecordMerger` implementation class from the list of classes in `hoodie.write.record.merge.custom.implementation.classes` that has the specified merge strategy ID.<br />`Config Param: RECORD_MERGE_STRATEGY_ID`<br />`Since Version: 0.13.0` |
+| hoodie.write.record.merge.custom.implementation.classes | N/A (Optional) | List of `HoodieRecordMerger` implementations constituting Hudi's merging strategy based on the engine used. Hudi picks the `HoodieRecordMerger` implementation class from this list based on the specified `hoodie.write.record.merge.strategy.id`.<br />`Config Param: RECORD_MERGE_IMPL_CLASSES`<br />`Since Version: 0.13.0` |
### Record Payloads
@@ -224,7 +233,7 @@ Result data after merging using `PartialUpdateAvroPayload`:
Payload class can be specified using the below configs. For more advanced configs refer [here](https://hudi.apache.org/docs/configurations#RECORD_PAYLOAD)
-**Spark based configs;**
+**Spark based configs:**
| Config Name | Default | Description |
| ---------------------------------------| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|