You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/16 21:45:58 UTC
[GitHub] [hudi] kumudkumartirupati opened a new pull request, #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
kumudkumartirupati opened a new pull request, #5598:
URL: https://github.com/apache/hudi/pull/5598
## What is the purpose of the pull request
Updates the missing content of DeltaStreamer documentation of v0.11.0.
## Brief change log
* Adds the missing `--enable-sync` config option
* Adds deprecation note for `--enable-hive-sync` config option.
* Adds the missing `--sync-tool-classes` config option that enables syncing to multiple meta providers. This option has parity with HoodieMultiTableDeltaStreamer as per the PR #5597
## Verify this pull request
This pull request is for updating the documentation.
## Committer checklist
- [x] Has a corresponding JIRA in PR title & commit
- [x] Commit message is descriptive of the change
- [ ] CI is green
- [x] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #5598:
URL: https://github.com/apache/hudi/pull/5598#issuecomment-1133904767
> @xushiyan I have updated the docs for 0.10.0 and 0.10.1 as well for the `--enable-sync` flag if this is what you meant. Please check and let me know.
@kumudkumartirupati i actually meant the "current" version, which is in sync with master version. The versioned docs are snapshotted for each release. You can read more from the README under `asf-site` branch to understand the versioning.
Once you land #5597 you can come back here and update according to the latest development.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kumudkumartirupati commented on pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
kumudkumartirupati commented on PR #5598:
URL: https://github.com/apache/hudi/pull/5598#issuecomment-1133844105
Also, @xushiyan `--sync-tool-classes` is not supposed to be present in `0.11.0`. I have removed it there and will re-add it once #5597 is finalized and the release version is known.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kumudkumartirupati commented on a diff in pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
kumudkumartirupati commented on code in PR #5598:
URL: https://github.com/apache/hudi/pull/5598#discussion_r875540353
##########
website/versioned_docs/version-0.11.0/hoodie_deltastreamer.md:
##########
@@ -48,8 +48,15 @@ Options:
Compaction is enabled for MoR table by default. This flag disables it
Default: false
--enable-hive-sync
- Enable syncing to hive
+ Enable syncing to hive (Deprecated in favor of --enable-sync and --sync-tool-classes)
Default: false
+ --enable-sync
+ Enable syncing meta
+ Default: false
+ --sync-tool-classes
+ Classes (comma-separated) to be used for syncing meta. Shall be used only when --enable-sync or --enable-hive-sync is set to true
+ Note: When used this with deprecated --enable-hive-sync flag, HiveSyncTool will always be run along with any other classes mentioned in here.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kumudkumartirupati commented on a diff in pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
kumudkumartirupati commented on code in PR #5598:
URL: https://github.com/apache/hudi/pull/5598#discussion_r875271120
##########
website/versioned_docs/version-0.11.0/hoodie_deltastreamer.md:
##########
@@ -48,8 +48,15 @@ Options:
Compaction is enabled for MoR table by default. This flag disables it
Default: false
--enable-hive-sync
- Enable syncing to hive
+ Enable syncing to hive (Deprecated in favor of --enable-sync and --sync-tool-classes)
Default: false
+ --enable-sync
+ Enable syncing meta
+ Default: false
+ --sync-tool-classes
+ Classes (comma-separated) to be used for syncing meta. Shall be used only when --enable-sync or --enable-hive-sync is set to true
+ Note: When used this with deprecated --enable-hive-sync flag, HiveSyncTool will always be run along with any other classes mentioned in here.
Review Comment:
Removed the note. Please check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kumudkumartirupati commented on pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
kumudkumartirupati commented on PR #5598:
URL: https://github.com/apache/hudi/pull/5598#issuecomment-1133839866
@xushiyan I have updated the docs for 0.10.0 and 0.10.1 as well for the `--enable-sync` flag if this is what you meant. Please check and let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kumud-hs commented on a diff in pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
kumud-hs commented on code in PR #5598:
URL: https://github.com/apache/hudi/pull/5598#discussion_r874501502
##########
website/versioned_docs/version-0.11.0/hoodie_deltastreamer.md:
##########
@@ -48,8 +48,15 @@ Options:
Compaction is enabled for MoR table by default. This flag disables it
Default: false
--enable-hive-sync
- Enable syncing to hive
+ Enable syncing to hive (Deprecated in favor of --enable-sync and --sync-tool-classes)
Default: false
+ --enable-sync
+ Enable syncing meta
+ Default: false
+ --sync-tool-classes
+ Classes (comma-separated) to be used for syncing meta. Shall be used only when --enable-sync or --enable-hive-sync is set to true
+ Note: When used this with deprecated --enable-hive-sync flag, HiveSyncTool will always be run along with any other classes mentioned in here.
Review Comment:
Did you mean to remove the `Note:` section?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a diff in pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on code in PR #5598:
URL: https://github.com/apache/hudi/pull/5598#discussion_r874484897
##########
website/versioned_docs/version-0.11.0/hoodie_deltastreamer.md:
##########
@@ -48,8 +48,15 @@ Options:
Compaction is enabled for MoR table by default. This flag disables it
Default: false
--enable-hive-sync
- Enable syncing to hive
+ Enable syncing to hive (Deprecated in favor of --enable-sync and --sync-tool-classes)
Default: false
+ --enable-sync
+ Enable syncing meta
+ Default: false
+ --sync-tool-classes
+ Classes (comma-separated) to be used for syncing meta. Shall be used only when --enable-sync or --enable-hive-sync is set to true
+ Note: When used this with deprecated --enable-hive-sync flag, HiveSyncTool will always be run along with any other classes mentioned in here.
Review Comment:
nit: remove this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kumudkumartirupati commented on a diff in pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
kumudkumartirupati commented on code in PR #5598:
URL: https://github.com/apache/hudi/pull/5598#discussion_r874504137
##########
website/versioned_docs/version-0.11.0/hoodie_deltastreamer.md:
##########
@@ -48,8 +48,15 @@ Options:
Compaction is enabled for MoR table by default. This flag disables it
Default: false
--enable-hive-sync
- Enable syncing to hive
+ Enable syncing to hive (Deprecated in favor of --enable-sync and --sync-tool-classes)
Default: false
+ --enable-sync
+ Enable syncing meta
+ Default: false
+ --sync-tool-classes
+ Classes (comma-separated) to be used for syncing meta. Shall be used only when --enable-sync or --enable-hive-sync is set to true
+ Note: When used this with deprecated --enable-hive-sync flag, HiveSyncTool will always be run along with any other classes mentioned in here.
Review Comment:
Did you mean to remove the `Note:` section?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
xushiyan merged PR #5598:
URL: https://github.com/apache/hudi/pull/5598
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] pratyakshsharma commented on a diff in pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
pratyakshsharma commented on code in PR #5598:
URL: https://github.com/apache/hudi/pull/5598#discussion_r875534409
##########
website/versioned_docs/version-0.11.0/hoodie_deltastreamer.md:
##########
@@ -48,8 +48,15 @@ Options:
Compaction is enabled for MoR table by default. This flag disables it
Default: false
--enable-hive-sync
- Enable syncing to hive
+ Enable syncing to hive (Deprecated in favor of --enable-sync and --sync-tool-classes)
Default: false
+ --enable-sync
+ Enable syncing meta
+ Default: false
+ --sync-tool-classes
+ Classes (comma-separated) to be used for syncing meta. Shall be used only when --enable-sync or --enable-hive-sync is set to true
+ Note: When used this with deprecated --enable-hive-sync flag, HiveSyncTool will always be run along with any other classes mentioned in here.
Review Comment:
No I meant to indicate this change -
When used this with -> When used with
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kumud-hs commented on a diff in pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
kumud-hs commented on code in PR #5598:
URL: https://github.com/apache/hudi/pull/5598#discussion_r875540103
##########
website/versioned_docs/version-0.11.0/hoodie_deltastreamer.md:
##########
@@ -48,8 +48,15 @@ Options:
Compaction is enabled for MoR table by default. This flag disables it
Default: false
--enable-hive-sync
- Enable syncing to hive
+ Enable syncing to hive (Deprecated in favor of --enable-sync and --sync-tool-classes)
Default: false
+ --enable-sync
+ Enable syncing meta
+ Default: false
+ --sync-tool-classes
+ Classes (comma-separated) to be used for syncing meta. Shall be used only when --enable-sync or --enable-hive-sync is set to true
+ Note: When used this with deprecated --enable-hive-sync flag, HiveSyncTool will always be run along with any other classes mentioned in here.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] kumudkumartirupati commented on pull request #5598: [HUDI-4107] Updated documentation for 0.11.0 - DeltaStreamer
Posted by GitBox <gi...@apache.org>.
kumudkumartirupati commented on PR #5598:
URL: https://github.com/apache/hudi/pull/5598#issuecomment-1134216851
> > @xushiyan I have updated the docs for 0.10.0 and 0.10.1 as well for the `--enable-sync` flag if this is what you meant. Please check and let me know.
>
> @kumudkumartirupati i actually meant the "current" version, which is in sync with master version. The versioned docs are snapshotted for each release. You can read more from the README under `asf-site` branch to understand the versioning.
>
> Once you land #5597 you can come back here and update according to the latest development.
Thanks for the pointers @xushiyan . I have done the necessary changes. I will update if any, post #5597 is landed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org