You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/24 06:28:29 UTC
[GitHub] [hudi] danny0405 opened a new pull request, #7296: [HUDI-5234] Streaming read skip clustering
danny0405 opened a new pull request, #7296:
URL: https://github.com/apache/hudi/pull/7296
### Change Logs
Supports skipping the clustering commits while reading. This can make the streaming read under clustering more efficient because the clustering files are all rewritten with the original records.
### Impact
No
### Risk level (write none, low medium or high below)
none
### Documentation Update
Should update the doc whith this new option `read.streaming.skip_clustering`.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #7296:
URL: https://github.com/apache/hudi/pull/7296
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7296:
URL: https://github.com/apache/hudi/pull/7296#issuecomment-1326049366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 245edcb8ea7ca11a2df3ce6119022257cba59cea UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7296:
URL: https://github.com/apache/hudi/pull/7296#issuecomment-1326230560
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13226",
"triggerID" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"triggerType" : "PUSH"
}, {
"hash" : "10079b795c1684862be7dbfdb167882c56bdc7cb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13229",
"triggerID" : "10079b795c1684862be7dbfdb167882c56bdc7cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 245edcb8ea7ca11a2df3ce6119022257cba59cea Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13226)
* 10079b795c1684862be7dbfdb167882c56bdc7cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13229)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7296:
URL: https://github.com/apache/hudi/pull/7296#issuecomment-1326214889
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13226",
"triggerID" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 245edcb8ea7ca11a2df3ce6119022257cba59cea Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13226)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7296:
URL: https://github.com/apache/hudi/pull/7296#discussion_r1031099300
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestData.java:
##########
@@ -429,6 +430,35 @@ public static void writeData(
funcWrapper.close();
}
+ /**
+ * Write a list of row data with Hoodie format base on the given configuration.
+ *
+ * @param dataBuffer The data buffer to write
+ * @param conf The flink configuration
+ * @throws Exception if error occurs
+ */
+ public static void writeDataAsInsert(
+ List<RowData> dataBuffer,
+ Configuration conf) throws Exception {
+ InsertFunctionWrapper<RowData> funcWrapper = new InsertFunctionWrapper<>(
Review Comment:
TODO: would refactor the code with writer wrapper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7296:
URL: https://github.com/apache/hudi/pull/7296#issuecomment-1326053923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13226",
"triggerID" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 245edcb8ea7ca11a2df3ce6119022257cba59cea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13226)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7296:
URL: https://github.com/apache/hudi/pull/7296#discussion_r1031287497
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -280,6 +280,14 @@ private FlinkOptions() {
+ "usually with delta time compaction strategy that is long enough, for e.g, one week;\n"
+ "2) changelog mode is enabled, this option is a solution to keep data integrity");
+ // this option is experimental
+ public static final ConfigOption<Boolean> READ_STREAMING_SKIP_CLUSTERING = ConfigOptions
+ .key("read.streaming.skip_clustering")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Whether to skip clustering instants for streaming read,\n"
+ + "to avoid reading duplicates");
Review Comment:
Yeah, that's needed, can you add more cases for clustering here ? You can add UTs in `TestInputFormat`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on a diff in pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #7296:
URL: https://github.com/apache/hudi/pull/7296#discussion_r1031289085
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -280,6 +280,14 @@ private FlinkOptions() {
+ "usually with delta time compaction strategy that is long enough, for e.g, one week;\n"
+ "2) changelog mode is enabled, this option is a solution to keep data integrity");
+ // this option is experimental
+ public static final ConfigOption<Boolean> READ_STREAMING_SKIP_CLUSTERING = ConfigOptions
+ .key("read.streaming.skip_clustering")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Whether to skip clustering instants for streaming read,\n"
+ + "to avoid reading duplicates");
Review Comment:
@danny0405, ok. I will do adding the UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] SteNicholas commented on a diff in pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
SteNicholas commented on code in PR #7296:
URL: https://github.com/apache/hudi/pull/7296#discussion_r1031205149
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -280,6 +280,14 @@ private FlinkOptions() {
+ "usually with delta time compaction strategy that is long enough, for e.g, one week;\n"
+ "2) changelog mode is enabled, this option is a solution to keep data integrity");
+ // this option is experimental
+ public static final ConfigOption<Boolean> READ_STREAMING_SKIP_CLUSTERING = ConfigOptions
+ .key("read.streaming.skip_clustering")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("Whether to skip clustering instants for streaming read,\n"
+ + "to avoid reading duplicates");
Review Comment:
Could you add the cases which cause reading duplicates in the description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7296:
URL: https://github.com/apache/hudi/pull/7296#issuecomment-1326223061
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13226",
"triggerID" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"triggerType" : "PUSH"
}, {
"hash" : "10079b795c1684862be7dbfdb167882c56bdc7cb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "10079b795c1684862be7dbfdb167882c56bdc7cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 245edcb8ea7ca11a2df3ce6119022257cba59cea Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13226)
* 10079b795c1684862be7dbfdb167882c56bdc7cb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7296: [HUDI-5234] Streaming read skip clustering
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7296:
URL: https://github.com/apache/hudi/pull/7296#issuecomment-1326397199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13226",
"triggerID" : "245edcb8ea7ca11a2df3ce6119022257cba59cea",
"triggerType" : "PUSH"
}, {
"hash" : "10079b795c1684862be7dbfdb167882c56bdc7cb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13229",
"triggerID" : "10079b795c1684862be7dbfdb167882c56bdc7cb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 10079b795c1684862be7dbfdb167882c56bdc7cb Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13229)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org