You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "huangxiaopingRD (via GitHub)" <gi...@apache.org> on 2023/03/14 07:10:31 UTC
[GitHub] [hudi] huangxiaopingRD opened a new pull request, #8175: [MINOR] Improve the description of operation in HoodieDeltaStreamer
huangxiaopingRD opened a new pull request, #8175:
URL: https://github.com/apache/hudi/pull/8175
### Change Logs
Improve the description of operation in HoodieDeltaStreamer
### Impact
No
### Risk level (write none, low medium or high below)
None
### Documentation Update
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8175: [MINOR] Improve the description of operation in HoodieDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8175:
URL: https://github.com/apache/hudi/pull/8175#issuecomment-1467552769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15707",
"triggerID" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 305d9c81a526b96c1c0cee2ad209e936596bf247 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15707)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8175: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8175:
URL: https://github.com/apache/hudi/pull/8175#issuecomment-1469212233
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15707",
"triggerID" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"triggerType" : "PUSH"
}, {
"hash" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15709",
"triggerID" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"triggerType" : "PUSH"
}, {
"hash" : "f6412c7743f9c832f48a788e626379fa2144697a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15723",
"triggerID" : "f6412c7743f9c832f48a788e626379fa2144697a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 752e603aa5164e743d0a7102c6ab8db99aa32905 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15709)
* f6412c7743f9c832f48a788e626379fa2144697a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15723)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8175: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8175:
URL: https://github.com/apache/hudi/pull/8175#issuecomment-1467630983
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15707",
"triggerID" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"triggerType" : "PUSH"
}, {
"hash" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 305d9c81a526b96c1c0cee2ad209e936596bf247 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15707)
* 752e603aa5164e743d0a7102c6ab8db99aa32905 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8175: [MINOR] Improve the description of operation in HoodieDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8175:
URL: https://github.com/apache/hudi/pull/8175#issuecomment-1467539633
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 305d9c81a526b96c1c0cee2ad209e936596bf247 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8175: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8175:
URL: https://github.com/apache/hudi/pull/8175#discussion_r1136775637
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java:
##########
@@ -268,8 +268,9 @@ public static class Config implements Serializable {
+ "Default: No limit, e.g: DFS-Source => max bytes to read, Kafka-Source => max events to read")
public long sourceLimit = Long.MAX_VALUE;
- @Parameter(names = {"--op"}, description = "Takes one of these values : UPSERT (default), INSERT (use when input "
- + "is purely new data/inserts to gain speed)", converter = OperationConverter.class)
+ @Parameter(names = {"--op"}, description = "Takes one of these values : UPSERT (default), INSERT, "
+ + "BULK_INSERT, INSERT_OVERWRITE, INSERT_OVERWRITE_TABLE, DELETE_PARTITION",
+ converter = OperationConverter.class)
Review Comment:
Delta streamer support `BULK_INSERT, INSERT_OVERWRITE, INSERT_OVERWRITE_TABLE, DELETE_PARTITION` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] huangxiaopingRD commented on a diff in pull request #8175: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer
Posted by "huangxiaopingRD (via GitHub)" <gi...@apache.org>.
huangxiaopingRD commented on code in PR #8175:
URL: https://github.com/apache/hudi/pull/8175#discussion_r1136796193
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java:
##########
@@ -268,8 +268,9 @@ public static class Config implements Serializable {
+ "Default: No limit, e.g: DFS-Source => max bytes to read, Kafka-Source => max events to read")
public long sourceLimit = Long.MAX_VALUE;
- @Parameter(names = {"--op"}, description = "Takes one of these values : UPSERT (default), INSERT (use when input "
- + "is purely new data/inserts to gain speed)", converter = OperationConverter.class)
+ @Parameter(names = {"--op"}, description = "Takes one of these values : UPSERT (default), INSERT, "
+ + "BULK_INSERT, INSERT_OVERWRITE, INSERT_OVERWRITE_TABLE, DELETE_PARTITION",
+ converter = OperationConverter.class)
Review Comment:
Yes, we can see the types already supported here.
https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #8175: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #8175:
URL: https://github.com/apache/hudi/pull/8175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8175: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8175:
URL: https://github.com/apache/hudi/pull/8175#issuecomment-1467646631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15707",
"triggerID" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"triggerType" : "PUSH"
}, {
"hash" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15709",
"triggerID" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 752e603aa5164e743d0a7102c6ab8db99aa32905 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15709)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8175: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8175:
URL: https://github.com/apache/hudi/pull/8175#issuecomment-1469364587
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15707",
"triggerID" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"triggerType" : "PUSH"
}, {
"hash" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15709",
"triggerID" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"triggerType" : "PUSH"
}, {
"hash" : "f6412c7743f9c832f48a788e626379fa2144697a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15723",
"triggerID" : "f6412c7743f9c832f48a788e626379fa2144697a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f6412c7743f9c832f48a788e626379fa2144697a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15723)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8175: [HUDI-5931] Improve the description of operation in HoodieDeltaStreamer
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8175:
URL: https://github.com/apache/hudi/pull/8175#issuecomment-1469206600
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15707",
"triggerID" : "305d9c81a526b96c1c0cee2ad209e936596bf247",
"triggerType" : "PUSH"
}, {
"hash" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15709",
"triggerID" : "752e603aa5164e743d0a7102c6ab8db99aa32905",
"triggerType" : "PUSH"
}, {
"hash" : "f6412c7743f9c832f48a788e626379fa2144697a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f6412c7743f9c832f48a788e626379fa2144697a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 752e603aa5164e743d0a7102c6ab8db99aa32905 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15709)
* f6412c7743f9c832f48a788e626379fa2144697a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org