You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/08/08 05:40:43 UTC
[GitHub] [hudi] wangxianghu opened a new pull request #1935: [HUDI-1121][DOC]Provide a document describing how to use callback
wangxianghu opened a new pull request #1935:
URL: https://github.com/apache/hudi/pull/1935
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*Add description of the config of write commit callback to describe how to use callback*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yanghua merged pull request #1935: [HUDI-1121][DOC]Provide a document describing how to use callback
Posted by GitBox <gi...@apache.org>.
yanghua merged pull request #1935:
URL: https://github.com/apache/hudi/pull/1935
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #1935: [HUDI-1121][DOC]Provide a document describing how to use callback
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1935:
URL: https://github.com/apache/hudi/pull/1935#discussion_r467454662
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -510,3 +510,28 @@ Property: `hoodie.memory.compaction.fraction` <br/>
#### withWriteStatusFailureFraction(failureFraction = 0.1) {#withWriteStatusFailureFraction}
Property: `hoodie.memory.writestatus.failure.fraction` <br/>
<span style="color:grey">This property controls what fraction of the failed record, exceptions we report back to driver</span>
+
+#### Write commit callback configs
+Controls callback behavior on write commit
Review comment:
> would describe in details, such as if enables the callback but failed to callback.
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #1935: [HUDI-1121][DOC]Provide a document describing how to use callback
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1935:
URL: https://github.com/apache/hudi/pull/1935#discussion_r467454643
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -510,3 +510,28 @@ Property: `hoodie.memory.compaction.fraction` <br/>
#### withWriteStatusFailureFraction(failureFraction = 0.1) {#withWriteStatusFailureFraction}
Property: `hoodie.memory.writestatus.failure.fraction` <br/>
<span style="color:grey">This property controls what fraction of the failed record, exceptions we report back to driver</span>
+
+#### Write commit callback configs
+Controls callback behavior on write commit
+[withCallbackConfig](#withCallbackConfig) (HoodieWriteCommitCallbackConfig) <br/>
+<span style="color:grey">Callback related configs</span>
+
+##### writeCommitCallbackOn(callbackOn = false) {#writeCommitCallbackOn}
+Property: `hoodie.write.commit.callback.on` <br/>
+<span style="color:grey">Turn callback on/off. off by default.</span>
+
+##### withCallbackClass(callbackClass) {#withCallbackClass}
+Property: `hoodie.write.commit.callback.class` <br/>
+<span style="color:grey">Full path of user-defined callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
+
+##### withCallbackHttpUrl(url) {#withCallbackHttpUrl}
Review comment:
> would add `#### HoodieWriteCommitHttpCallback` before the following configs since the configs belong to HoodieWriteCommitHttpCallback
yes, my mistake.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1935: [HUDI-1121][DOC]Provide a document describing how to use callback
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1935:
URL: https://github.com/apache/hudi/pull/1935#discussion_r467437255
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -510,3 +510,28 @@ Property: `hoodie.memory.compaction.fraction` <br/>
#### withWriteStatusFailureFraction(failureFraction = 0.1) {#withWriteStatusFailureFraction}
Property: `hoodie.memory.writestatus.failure.fraction` <br/>
<span style="color:grey">This property controls what fraction of the failed record, exceptions we report back to driver</span>
+
+#### Write commit callback configs
+Controls callback behavior on write commit
Review comment:
would describe in details, such as if enables the callback but failed to callback.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1935: [HUDI-1121][DOC]Provide a document describing how to use callback
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1935:
URL: https://github.com/apache/hudi/pull/1935#discussion_r467436612
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -510,3 +510,28 @@ Property: `hoodie.memory.compaction.fraction` <br/>
#### withWriteStatusFailureFraction(failureFraction = 0.1) {#withWriteStatusFailureFraction}
Property: `hoodie.memory.writestatus.failure.fraction` <br/>
<span style="color:grey">This property controls what fraction of the failed record, exceptions we report back to driver</span>
+
+#### Write commit callback configs
+Controls callback behavior on write commit
+[withCallbackConfig](#withCallbackConfig) (HoodieWriteCommitCallbackConfig) <br/>
+<span style="color:grey">Callback related configs</span>
+
+##### writeCommitCallbackOn(callbackOn = false) {#writeCommitCallbackOn}
+Property: `hoodie.write.commit.callback.on` <br/>
+<span style="color:grey">Turn callback on/off. off by default.</span>
+
+##### withCallbackClass(callbackClass) {#withCallbackClass}
+Property: `hoodie.write.commit.callback.class` <br/>
+<span style="color:grey">Full path of user-defined callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
+
+##### withCallbackHttpUrl(url) {#withCallbackHttpUrl}
Review comment:
would add `#### HoodieWriteCommitHttpCallback` before the following configs since the configs belong to HoodieWriteCommitHttpCallback
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #1935: [HUDI-1121][DOC]Provide a document describing how to use callback
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #1935:
URL: https://github.com/apache/hudi/pull/1935#discussion_r467459307
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -510,3 +510,30 @@ Property: `hoodie.memory.compaction.fraction` <br/>
#### withWriteStatusFailureFraction(failureFraction = 0.1) {#withWriteStatusFailureFraction}
Property: `hoodie.memory.writestatus.failure.fraction` <br/>
<span style="color:grey">This property controls what fraction of the failed record, exceptions we report back to driver</span>
+
+### Write commit callback configs
+Controls callback behavior on write commit. Exception will be thrown if user enabled the callback service and errors occurred during the process of callback. Currently support http callback only, kafka implementation will be supported in the near future.
+[withCallbackConfig](#withCallbackConfig) (HoodieWriteCommitCallbackConfig) <br/>
+<span style="color:grey">Callback related configs</span>
+
+#### HoodieWriteCommitHttpCallback
+
+##### writeCommitCallbackOn(callbackOn = false) {#writeCommitCallbackOn}
+Property: `hoodie.write.commit.callback.on` <br/>
+<span style="color:grey">Turn callback on/off. off by default.</span>
+
+##### withCallbackClass(callbackClass) {#withCallbackClass}
+Property: `hoodie.write.commit.callback.class` <br/>
+<span style="color:grey">Full path of user-defined callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
Review comment:
> move into Write commit callback configs section would be better?
yes, more clear
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #1935: [HUDI-1121][DOC]Provide a document describing how to use callback
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #1935:
URL: https://github.com/apache/hudi/pull/1935#issuecomment-670828182
@yanghua @leesf please take a look when free
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #1935: [HUDI-1121][DOC]Provide a document describing how to use callback
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #1935:
URL: https://github.com/apache/hudi/pull/1935#discussion_r467456970
##########
File path: docs/_docs/2_4_configurations.md
##########
@@ -510,3 +510,30 @@ Property: `hoodie.memory.compaction.fraction` <br/>
#### withWriteStatusFailureFraction(failureFraction = 0.1) {#withWriteStatusFailureFraction}
Property: `hoodie.memory.writestatus.failure.fraction` <br/>
<span style="color:grey">This property controls what fraction of the failed record, exceptions we report back to driver</span>
+
+### Write commit callback configs
+Controls callback behavior on write commit. Exception will be thrown if user enabled the callback service and errors occurred during the process of callback. Currently support http callback only, kafka implementation will be supported in the near future.
+[withCallbackConfig](#withCallbackConfig) (HoodieWriteCommitCallbackConfig) <br/>
+<span style="color:grey">Callback related configs</span>
+
+#### HoodieWriteCommitHttpCallback
+
+##### writeCommitCallbackOn(callbackOn = false) {#writeCommitCallbackOn}
+Property: `hoodie.write.commit.callback.on` <br/>
+<span style="color:grey">Turn callback on/off. off by default.</span>
+
+##### withCallbackClass(callbackClass) {#withCallbackClass}
+Property: `hoodie.write.commit.callback.class` <br/>
+<span style="color:grey">Full path of user-defined callback class and must be a subclass of HoodieWriteCommitCallback class, org.apache.hudi.callback.impl.HoodieWriteCommitHttpCallback by default</span>
Review comment:
move into Write commit callback configs section would be better?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org