You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by vi...@apache.org on 2020/08/22 13:48:25 UTC
[hudi] branch asf-site updated: [HUDI-1186][DOC] Add description of
write commit callback by kafka to document (#1974)
This is an automated email from the ASF dual-hosted git repository.
vinoyang pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new ce51638 [HUDI-1186][DOC] Add description of write commit callback by kafka to document (#1974)
ce51638 is described below
commit ce51638a434a40bc50a8f7ae64b8fce00ad98817
Author: Mathieu <wx...@126.com>
AuthorDate: Sat Aug 22 21:48:18 2020 +0800
[HUDI-1186][DOC] Add description of write commit callback by kafka to document (#1974)
---
docs/_docs/2_4_configurations.cn.md | 31 ++++++++++++++++++++++++++++---
docs/_docs/2_4_configurations.md | 30 +++++++++++++++++++++++++++---
2 files changed, 55 insertions(+), 6 deletions(-)
diff --git a/docs/_docs/2_4_configurations.cn.md b/docs/_docs/2_4_configurations.cn.md
index 8578285..9436971 100644
--- a/docs/_docs/2_4_configurations.cn.md
+++ b/docs/_docs/2_4_configurations.cn.md
@@ -549,7 +549,7 @@ Hudi提供了一个选项，可以通过将对该分区中的插入作为对现
<span style="color:grey">此属性控制报告给驱动程序的失败记录和异常的比例</span>
### 写提交回调配置
-控制写提交的回调。 如果用户启用了回调并且回调过程发生了错误，则会抛出异常。 当前只支持Http回调方式，Kafka不久后会支持。
+控制写提交的回调。 如果用户启用了回调并且回调过程发生了错误，则会抛出异常。 当前支持HTTP, Kafka 两种回调方式。
[withCallbackConfig](#withCallbackConfig) (HoodieWriteCommitCallbackConfig) <br/>
<span style="color:grey">写提交回调相关配置</span>
@@ -559,9 +559,10 @@ Property: `hoodie.write.commit.callback.on` <br/>
##### withCallbackClass(callbackClass) {#withCallbackClass}
Property: `hoodie.write.commit.callback.class` <br/>
-<span style="color:grey">用户自定义回调的类全路径名，回调类必须为HoodieWriteCommitCallback的子类。默认 org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback</span>
+<span style="color:grey">回调类的完全限定名，必须实现HoodieWriteCommitCallback接口。默认 org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback</span>
-#### HoodieWriteCommitHttpCallback
+#### HTTP CALLBACK
+通过 HTTP 发送写提交回调信息. 这是默认的实现方式，用户不需要显式指定。
##### withCallbackHttpUrl(url) {#withCallbackHttpUrl}
Property: `hoodie.write.commit.callback.http.url` <br/>
@@ -574,3 +575,27 @@ Property: `hoodie.write.commit.callback.http.timeout.seconds` <br/>
##### withCallbackHttpApiKey(apiKey) {#withCallbackHttpApiKey}
Property: `hoodie.write.commit.callback.http.api.key` <br/>
<span style="color:grey">Http 回调秘钥. 默认 hudi_write_commit_http_callback</span>
+
+#### KAFKA CALLBACK
+使用Kafka发送写提交回调信息, 用户需要配置 `hoodie.write.commit.callback.class` = `org.apache.hudi.utilities.callback.kafka.HoodieWriteCommitKafkaCallback`
+
+##### CALLBACK_KAFKA_BOOTSTRAP_SERVERS
+Property: `hoodie.write.commit.callback.kafka.bootstrap.servers` <br/>
+<span style="color:grey">Kafka 集群地址</span>
+
+##### CALLBACK_KAFKA_TOPIC
+Property: `hoodie.write.commit.callback.kafka.topic` <br/>
+<span style="color:grey">发送回调信息的topic</span>
+
+##### CALLBACK_KAFKA_PARTITION
+Property: `hoodie.write.commit.callback.kafka.partition` <br/>
+<span style="color:grey">指定发送的分区, 默认 0 </span>
+
+##### CALLBACK_KAFKA_ACKS
+Property: `hoodie.write.commit.callback.kafka.acks` <br/>
+<span style="color:grey">Acks 级别, 默认 `all`</span>
+
+##### CALLBACK_KAFKA_RETRIES
+Property: `hoodie.write.commit.callback.kafka.retries` <br/>
+<span style="color:grey">Kafka 发送数据失败重试次数. 默认 3 次</span>
+
diff --git a/docs/_docs/2_4_configurations.md b/docs/_docs/2_4_configurations.md
index 47bd08c..5536ac0 100644
--- a/docs/_docs/2_4_configurations.md
+++ b/docs/_docs/2_4_configurations.md
@@ -512,7 +512,7 @@ Property: `hoodie.memory.writestatus.failure.fraction` <br/>
<span style="color:grey">This property controls what fraction of the failed record, exceptions we report back to driver</span>
### Write commit callback configs
-Controls callback behavior on write commit. Exception will be thrown if user enabled the callback service and errors occurred during the process of callback. Currently support http callback only, kafka implementation will be supported in the near future.
+Controls callback behavior on write commit. Exception will be thrown if user enabled the callback service and errors occurred during the process of callback. Currently support HTTP, Kafka type.
[withCallbackConfig](#withCallbackConfig) (HoodieWriteCommitCallbackConfig) <br/>
<span style="color:grey">Callback related configs</span>
@@ -522,9 +522,10 @@ Property: `hoodie.write.commit.callback.on` <br/>
##### withCallbackClass(callbackClass) {#withCallbackClass}
Property: `hoodie.write.commit.callback.class` <br/>
-<span style="color:grey">Full path of user-defined callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
+<span style="color:grey">Full path of callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
-#### HoodieWriteCommitHttpCallback
+#### HTTP CALLBACK
+Callback via HTTP, User does not need to specify this way explicitly, it is the default type.
##### withCallbackHttpUrl(url) {#withCallbackHttpUrl}
Property: `hoodie.write.commit.callback.http.url` <br/>
@@ -537,3 +538,26 @@ Property: `hoodie.write.commit.callback.http.timeout.seconds` <br/>
##### withCallbackHttpApiKey(apiKey) {#withCallbackHttpApiKey}
Property: `hoodie.write.commit.callback.http.api.key` <br/>
<span style="color:grey">Http callback API key. hudi_write_commit_http_callback by default</span>
+
+#### KAFKA CALLBACK
+To use kafka callback, User should set `hoodie.write.commit.callback.class` = `org.apache.hudi.utilities.callback.kafka.HoodieWriteCommitKafkaCallback`
+
+##### CALLBACK_KAFKA_BOOTSTRAP_SERVERS
+Property: `hoodie.write.commit.callback.kafka.bootstrap.servers` <br/>
+<span style="color:grey">Bootstrap servers of kafka callback cluster</span>
+
+##### CALLBACK_KAFKA_TOPIC
+Property: `hoodie.write.commit.callback.kafka.topic` <br/>
+<span style="color:grey">Kafka topic to be sent along with callback messages</span>
+
+##### CALLBACK_KAFKA_PARTITION
+Property: `hoodie.write.commit.callback.kafka.partition` <br/>
+<span style="color:grey">partition of `CALLBACK_KAFKA_TOPIC`, 0 by default</span>
+
+##### CALLBACK_KAFKA_ACKS
+Property: `hoodie.write.commit.callback.kafka.acks` <br/>
+<span style="color:grey">kafka acks level, `all` by default</span>
+
+##### CALLBACK_KAFKA_RETRIES
+Property: `hoodie.write.commit.callback.kafka.retries` <br/>
+<span style="color:grey">Times to retry. 3 by default</span>