You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by da...@apache.org on 2024/07/27 01:03:04 UTC
(hudi) branch asf-site updated: [HUDI-6191][DOCS] Improve passing the debezium checkpoint values to start job from offset (#11690)
This is an automated email from the ASF dual-hosted git repository.
danny0405 pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 08032842fbf [HUDI-6191][DOCS] Improve passing the debezium checkpoint values to start job from offset (#11690)
08032842fbf is described below
commit 08032842fbf195d5e848064f42aa7287b68379a2
Author: Vova Kolmakov <wo...@gmail.com>
AuthorDate: Sat Jul 27 08:02:57 2024 +0700
[HUDI-6191][DOCS] Improve passing the debezium checkpoint values to start job from offset (#11690)
Co-authored-by: Vova Kolmakov <ko...@huawei-partners.com>
---
website/docs/configurations.md | 4 ++--
website/docs/hoodie_streaming_ingestion.md | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/website/docs/configurations.md b/website/docs/configurations.md
index a6814502cdc..30bdd3e33c6 100644
--- a/website/docs/configurations.md
+++ b/website/docs/configurations.md
@@ -5,7 +5,7 @@ permalink: /docs/configurations.html
summary: This page covers the different ways of configuring your job to write/read Hudi tables. At a high level, you can control behaviour at few levels.
toc_min_heading_level: 2
toc_max_heading_level: 4
-last_modified_at: 2024-07-01T15:09:57.588
+last_modified_at: 2024-07-26T12:54:21.684
---
@@ -1996,7 +1996,7 @@ Configurations controlling the behavior of Kafka source in Hudi Streamer.
| [hoodie.streamer.source.kafka.value.deserializer.schema](#hoodiestreamersourcekafkavaluedeserializerschema) | (N/A) | Schema to deserialize the records.<br />`Config Param: KAFKA_VALUE_DESERIALIZER_SCHEMA` [...]
| [auto.offset.reset](#autooffsetreset) | LATEST | Kafka consumer strategy for reading data.<br />`Config Param: KAFKA_AUTO_OFFSET_RESET` [...]
| [hoodie.streamer.kafka.source.maxEvents](#hoodiestreamerkafkasourcemaxEvents) | 5000000 | Maximum number of records obtained in each batch.<br />`Config Param: MAX_EVENTS_FROM_KAFKA_SOURCE` [...]
-| [hoodie.streamer.source.kafka.checkpoint.type](#hoodiestreamersourcekafkacheckpointtype) | string | Kafka checkpoint type.<br />`Config Param: KAFKA_CHECKPOINT_TYPE` [...]
+| [hoodie.streamer.source.kafka.checkpoint.type](#hoodiestreamersourcekafkacheckpointtype) | string | Kafka checkpoint type. Value must be one of the following: string, timestamp, single_offset. Default type is string. For type string, checkpoint should be provided as: topicName,0:offset0,1:offset1,2:offset2. For type timestamp, checkpoint should be provided as long value of desired timestamp. For type single_offset, we as [...]
| [hoodie.streamer.source.kafka.enable.commit.offset](#hoodiestreamersourcekafkaenablecommitoffset) | false | Automatically submits offset to kafka.<br />`Config Param: ENABLE_KAFKA_COMMIT_OFFSET` [...]
| [hoodie.streamer.source.kafka.enable.failOnDataLoss](#hoodiestreamersourcekafkaenablefailOnDataLoss) | false | Fail when checkpoint goes out of bounds instead of seeking to earliest offsets.<br />`Config Param: ENABLE_FAIL_ON_DATA_LOSS` [...]
| [hoodie.streamer.source.kafka.fetch_partition.time.out](#hoodiestreamersourcekafkafetch_partitiontimeout) | 300000 | Time out for fetching partitions. 5min by default<br />`Config Param: KAFKA_FETCH_PARTITION_TIME_OUT` [...]
diff --git a/website/docs/hoodie_streaming_ingestion.md b/website/docs/hoodie_streaming_ingestion.md
index 6d3dc3a34e5..dca65b9b426 100644
--- a/website/docs/hoodie_streaming_ingestion.md
+++ b/website/docs/hoodie_streaming_ingestion.md
@@ -313,7 +313,7 @@ Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`.
If you need to change the checkpoints for reprocessing or replaying data you can use the following options:
-- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint.
+- `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](/docs/configurations#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a s [...]
- `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read.
For Kafka, this is the max # of events to read.