You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/08/17 06:59:30 UTC
[GitHub] [hudi] wangxianghu opened a new pull request #1974: [HUDI-1186]Add description of write commit callback by kafka to document
wangxianghu opened a new pull request #1974:
URL: https://github.com/apache/hudi/pull/1974
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*Add description of write commit callback by kafka to document*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #1974: [HUDI-1186][DOC]Add description of write commit callback by kafka to document
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1974:
URL: https://github.com/apache/hudi/pull/1974#discussion_r474677777
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -522,9 +522,10 @@ Property: `hoodie.write.commit.callback.on` <br/>
##### withCallbackClass(callbackClass) {#withCallbackClass}
Property: `hoodie.write.commit.callback.class` <br/>
-<span style="color:grey">Full path of user-defined callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
+<span style="color:grey">Full path of callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
-#### HoodieWriteCommitHttpCallback
+#### HTTP CALLBACK
+Callback via http, User does not need to specify this way explicitly, it is the default type.
Review comment:
> `http` -> `HTTP`
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #1974: [HUDI-1186][DOC]Add description of write commit callback by kafka to document
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1974:
URL: https://github.com/apache/hudi/pull/1974#discussion_r474638391
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -522,9 +522,10 @@ Property: `hoodie.write.commit.callback.on` <br/>
##### withCallbackClass(callbackClass) {#withCallbackClass}
Property: `hoodie.write.commit.callback.class` <br/>
-<span style="color:grey">Full path of user-defined callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
+<span style="color:grey">Full path of callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
-#### HoodieWriteCommitHttpCallback
+#### HTTP CALLBACK
+Callback via http, User does not need to specify this way explicitly, it is the default type.
Review comment:
`http` -> `HTTP`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #1974: [HUDI-1186][DOC]Add description of write commit callback by kafka to document
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1974:
URL: https://github.com/apache/hudi/pull/1974#discussion_r474401044
##########
File path: docs/_docs/2_4_configurations.cn.md
##########
@@ -549,7 +549,7 @@ Hudi提供了一个选项，可以通过将对该分区中的插入作为对现
<span style="color:grey">此属性控制报告给驱动程序的失败记录和异常的比例</span>
### 写提交回调配置
-控制写提交的回调。 如果用户启用了回调并且回调过程发生了错误，则会抛出异常。 当前只支持Http回调方式，Kafka不久后会支持。
+控制写提交的回调。 如果用户启用了回调并且回调过程发生了错误，则会抛出异常。 当前只支持 Http, Kafka 两种回调方式。
Review comment:
> Can we remove `只`? And `Http` -> `HTTP`?
done
##########
File path: docs/_docs/2_4_configurations.cn.md
##########
@@ -559,9 +559,10 @@ Property: `hoodie.write.commit.callback.on` <br/>
##### withCallbackClass(callbackClass) {#withCallbackClass}
Property: `hoodie.write.commit.callback.class` <br/>
-<span style="color:grey">用户自定义回调的类全路径名，回调类必须为HoodieWriteCommitCallback的子类。默认 org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback</span>
+<span style="color:grey">回调类全路径名，回调类必须为HoodieWriteCommitCallback的子类。默认 org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback</span>
Review comment:
> `回调类全路径名` -> `回调类的完全限定名` and `必须为HoodieWriteCommitCallback的子类` -> `必须实现HoodieWriteCommitCallback接口`
done
##########
File path: docs/_docs/2_4_configurations.cn.md
##########
@@ -574,3 +575,27 @@ Property: `hoodie.write.commit.callback.http.timeout.seconds` <br/>
##### withCallbackHttpApiKey(apiKey) {#withCallbackHttpApiKey}
Property: `hoodie.write.commit.callback.http.api.key` <br/>
<span style="color:grey">Http 回调秘钥. 默认 hudi_write_commit_http_callback</span>
+
+#### KAFKA CALLBACK
+使用Kafka发送写提交回调信息, 用户需要配置 `hoodie.write.commit.callback.class` = `org.apache.hudi.utilities.callback.kafka.HoodieWriteCommitKafkaCallback`
+
+##### CALLBACK_KAFKA_BOOTSTRAP_SERVERS
+Property: `hoodie.write.commit.callback.kafka.bootstrap.servers` <br/>
+<span style="color:grey">Kafka 集群地址</span>
+
+##### CALLBACK_KAFKA_TOPIC
+Property: `hoodie.write.commit.callback.kafka.topic` <br/>
+<span style="color:grey">发送回调信息的主题</span>
Review comment:
> `主题` -> topic
done
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -512,7 +512,7 @@ Property: `hoodie.memory.writestatus.failure.fraction` <br/>
<span style="color:grey">This property controls what fraction of the failed record, exceptions we report back to driver</span>
### Write commit callback configs
-Controls callback behavior on write commit. Exception will be thrown if user enabled the callback service and errors occurred during the process of callback. Currently support http callback only, kafka implementation will be supported in the near future.
+Controls callback behavior on write commit. Exception will be thrown if user enabled the callback service and errors occurred during the process of callback. Currently support http, kafka type only.
Review comment:
> Remove `only`.
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #1974: [HUDI-1186][DOC] Add description of write commit callback by kafka to document
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #1974:
URL: https://github.com/apache/hudi/pull/1974
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua commented on a change in pull request #1974: [HUDI-1186][DOC]Add description of write commit callback by kafka to document
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #1974:
URL: https://github.com/apache/hudi/pull/1974#discussion_r474347486
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -512,7 +512,7 @@ Property: `hoodie.memory.writestatus.failure.fraction` <br/>
<span style="color:grey">This property controls what fraction of the failed record, exceptions we report back to driver</span>
### Write commit callback configs
-Controls callback behavior on write commit. Exception will be thrown if user enabled the callback service and errors occurred during the process of callback. Currently support http callback only, kafka implementation will be supported in the near future.
+Controls callback behavior on write commit. Exception will be thrown if user enabled the callback service and errors occurred during the process of callback. Currently support http, kafka type only.
Review comment:
Remove `only`.
##########
File path: docs/_docs/2_4_configurations.cn.md
##########
@@ -574,3 +575,27 @@ Property: `hoodie.write.commit.callback.http.timeout.seconds` <br/>
##### withCallbackHttpApiKey(apiKey) {#withCallbackHttpApiKey}
Property: `hoodie.write.commit.callback.http.api.key` <br/>
<span style="color:grey">Http 回调秘钥. 默认 hudi_write_commit_http_callback</span>
+
+#### KAFKA CALLBACK
+使用Kafka发送写提交回调信息, 用户需要配置 `hoodie.write.commit.callback.class` = `org.apache.hudi.utilities.callback.kafka.HoodieWriteCommitKafkaCallback`
+
+##### CALLBACK_KAFKA_BOOTSTRAP_SERVERS
+Property: `hoodie.write.commit.callback.kafka.bootstrap.servers` <br/>
+<span style="color:grey">Kafka 集群地址</span>
+
+##### CALLBACK_KAFKA_TOPIC
+Property: `hoodie.write.commit.callback.kafka.topic` <br/>
+<span style="color:grey">发送回调信息的主题</span>
Review comment:
`主题` -> topic
##########
File path: docs/_docs/2_4_configurations.cn.md
##########
@@ -559,9 +559,10 @@ Property: `hoodie.write.commit.callback.on` <br/>
##### withCallbackClass(callbackClass) {#withCallbackClass}
Property: `hoodie.write.commit.callback.class` <br/>
-<span style="color:grey">用户自定义回调的类全路径名，回调类必须为HoodieWriteCommitCallback的子类。默认 org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback</span>
+<span style="color:grey">回调类全路径名，回调类必须为HoodieWriteCommitCallback的子类。默认 org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback</span>
Review comment:
`回调类全路径名` -> `回调类的完全限定名` and `必须为HoodieWriteCommitCallback的子类` -> `必须实现HoodieWriteCommitCallback接口`
##########
File path: docs/_docs/2_4_configurations.cn.md
##########
@@ -549,7 +549,7 @@ Hudi提供了一个选项，可以通过将对该分区中的插入作为对现
<span style="color:grey">此属性控制报告给驱动程序的失败记录和异常的比例</span>
### 写提交回调配置
-控制写提交的回调。 如果用户启用了回调并且回调过程发生了错误，则会抛出异常。 当前只支持Http回调方式，Kafka不久后会支持。
+控制写提交的回调。 如果用户启用了回调并且回调过程发生了错误，则会抛出异常。 当前只支持 Http, Kafka 两种回调方式。
Review comment:
Can we remove `只`? And `Http` -> `HTTP`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #1974: [HUDI-1186]Add description of write commit callback by kafka to document
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #1974:
URL: https://github.com/apache/hudi/pull/1974#issuecomment-674701428
@yanghua please review this after https://github.com/apache/hudi/pull/1886 is merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org