You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/21 08:20:25 UTC
[GitHub] [hudi] chenshzh opened a new pull request, #7017: [HUDI-5066] Support flink hoodie source metaclient cache
chenshzh opened a new pull request, #7017:
URL: https://github.com/apache/hudi/pull/7017
### Change Logs
Flink Table Planner will invoke `HoodieTableSource.copy()` when applying its `RelOptRule` such as
`PushPartitionIntoTableSourceScanRule`, thus result in multiple meta client instantiaton, which will
affect the performance of starting job quickly.
Here we promote that meta client might be cached to reused.
### Impact
Speed up flink sql job start by avoiding unnecessary meta client repeated creation.
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1291541628
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1291517242",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenshzh commented on a diff in pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
chenshzh commented on code in PR #7017:
URL: https://github.com/apache/hudi/pull/7017#discussion_r1006389908
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##########
@@ -162,7 +163,7 @@ public HoodieTableSource(
this.limit = limit == null ? NO_LIMIT_CONSTANT : limit;
this.filters = filters == null ? Collections.emptyList() : filters;
this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
- this.metaClient = StreamerUtil.metaClientForReader(conf, hadoopConf);
+ this.metaClient = Optional.ofNullable(metaClient).orElse(StreamerUtil.metaClientForReader(conf, hadoopConf));
Review Comment:
> initialization lazy
Here we have some more consideration:
During planned and optimized **metaclient** or **fileindex** might be used by different rules' **hoodie source** copy instance for multi times.
For example, actually **fileindex** is used in **PushPartitionIntoTableSourceScanRule** and **PushFilterIntoSourceScanRuleBase**
So it seems that initialization lazy will also cause the multi construction ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yuzhaojing commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
yuzhaojing commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1290274555
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1303574212
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1291517242",
"triggerType" : "MANUAL"
}, {
"hash" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78c0d932bdc06ab1b28d408d3b04d0a37bc160a7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1290331488
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1303589195
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1291517242",
"triggerType" : "MANUAL"
}, {
"hash" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12792",
"triggerID" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78c0d932bdc06ab1b28d408d3b04d0a37bc160a7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenshzh commented on a diff in pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
chenshzh commented on code in PR #7017:
URL: https://github.com/apache/hudi/pull/7017#discussion_r1015033559
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##########
@@ -162,7 +163,7 @@ public HoodieTableSource(
this.limit = limit == null ? NO_LIMIT_CONSTANT : limit;
this.filters = filters == null ? Collections.emptyList() : filters;
this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
- this.metaClient = StreamerUtil.metaClientForReader(conf, hadoopConf);
+ this.metaClient = Optional.ofNullable(metaClient).orElse(StreamerUtil.metaClientForReader(conf, hadoopConf));
Review Comment:
> Can we just decide whether it is null directly instead of using `Optional.ofNullable(metaClient).orElse` here ?
@danny0405 updated, pls review it one more time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1286634806
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bef3d62c19bac652daf1c3cd351adb06503a1d17 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1286698058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bef3d62c19bac652daf1c3cd351adb06503a1d17 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenshzh commented on a diff in pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
chenshzh commented on code in PR #7017:
URL: https://github.com/apache/hudi/pull/7017#discussion_r1014036249
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##########
@@ -162,7 +163,7 @@ public HoodieTableSource(
this.limit = limit == null ? NO_LIMIT_CONSTANT : limit;
this.filters = filters == null ? Collections.emptyList() : filters;
this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
- this.metaClient = StreamerUtil.metaClientForReader(conf, hadoopConf);
+ this.metaClient = Optional.ofNullable(metaClient).orElse(StreamerUtil.metaClientForReader(conf, hadoopConf));
Review Comment:
@danny0405 are there any other problems for this pr?
Have seen one pr: HUDI-5147 approved to solve he `fileindex` problems, which is part of the contents mentioned above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1305209285
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1291517242",
"triggerType" : "MANUAL"
}, {
"hash" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12792",
"triggerID" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"triggerType" : "PUSH"
}, {
"hash" : "836ce4252e35c00d246ea4e1097d4ce5bdfc4a91",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12823",
"triggerID" : "836ce4252e35c00d246ea4e1097d4ce5bdfc4a91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 836ce4252e35c00d246ea4e1097d4ce5bdfc4a91 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12823)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenshzh commented on a diff in pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
chenshzh commented on code in PR #7017:
URL: https://github.com/apache/hudi/pull/7017#discussion_r1006389908
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##########
@@ -162,7 +163,7 @@ public HoodieTableSource(
this.limit = limit == null ? NO_LIMIT_CONSTANT : limit;
this.filters = filters == null ? Collections.emptyList() : filters;
this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
- this.metaClient = StreamerUtil.metaClientForReader(conf, hadoopConf);
+ this.metaClient = Optional.ofNullable(metaClient).orElse(StreamerUtil.metaClientForReader(conf, hadoopConf));
Review Comment:
> initialization lazy
Here we have some more consideration:
During planned and optimized **metaclient** or **fileindex** might be used by different rules' **hoodie source** copy instances for multi times.
For example, actually **fileindex** is used in **PushPartitionIntoTableSourceScanRule** and **PushFilterIntoSourceScanRuleBase**
So it seems that initialization lazy will also cause the multi construction ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1286862142
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bef3d62c19bac652daf1c3cd351adb06503a1d17 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1290323791
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0000 Unknown: [CANCELED](TBD)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenshzh commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
chenshzh commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1290258456
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chenshzh commented on a diff in pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
chenshzh commented on code in PR #7017:
URL: https://github.com/apache/hudi/pull/7017#discussion_r1007944652
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##########
@@ -162,7 +163,7 @@ public HoodieTableSource(
this.limit = limit == null ? NO_LIMIT_CONSTANT : limit;
this.filters = filters == null ? Collections.emptyList() : filters;
this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
- this.metaClient = StreamerUtil.metaClientForReader(conf, hadoopConf);
+ this.metaClient = Optional.ofNullable(metaClient).orElse(StreamerUtil.metaClientForReader(conf, hadoopConf));
Review Comment:
@danny0405 shall we need to do more improvement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7017:
URL: https://github.com/apache/hudi/pull/7017#discussion_r1014547525
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##########
@@ -162,7 +163,7 @@ public HoodieTableSource(
this.limit = limit == null ? NO_LIMIT_CONSTANT : limit;
this.filters = filters == null ? Collections.emptyList() : filters;
this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
- this.metaClient = StreamerUtil.metaClientForReader(conf, hadoopConf);
+ this.metaClient = Optional.ofNullable(metaClient).orElse(StreamerUtil.metaClientForReader(conf, hadoopConf));
Review Comment:
Can we just decide whether it is null directly instead of using `Optional.ofNullable(metaClient).orElse` here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #7017:
URL: https://github.com/apache/hudi/pull/7017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1291537900
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1291517242",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] XuQianJin-Stars commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1291517242
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1291497972
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7017:
URL: https://github.com/apache/hudi/pull/7017#discussion_r1006349333
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java:
##########
@@ -162,7 +163,7 @@ public HoodieTableSource(
this.limit = limit == null ? NO_LIMIT_CONSTANT : limit;
this.filters = filters == null ? Collections.emptyList() : filters;
this.hadoopConf = HadoopConfigurations.getHadoopConf(conf);
- this.metaClient = StreamerUtil.metaClientForReader(conf, hadoopConf);
+ this.metaClient = Optional.ofNullable(metaClient).orElse(StreamerUtil.metaClientForReader(conf, hadoopConf));
Review Comment:
Another choice is to make `metaClient` and `fileIndex` initialization lazy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1304012173
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1291517242",
"triggerType" : "MANUAL"
}, {
"hash" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12792",
"triggerID" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78c0d932bdc06ab1b28d408d3b04d0a37bc160a7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12792)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1305047229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1291517242",
"triggerType" : "MANUAL"
}, {
"hash" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12792",
"triggerID" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"triggerType" : "PUSH"
}, {
"hash" : "836ce4252e35c00d246ea4e1097d4ce5bdfc4a91",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "836ce4252e35c00d246ea4e1097d4ce5bdfc4a91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78c0d932bdc06ab1b28d408d3b04d0a37bc160a7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12792)
* 836ce4252e35c00d246ea4e1097d4ce5bdfc4a91 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7017: [HUDI-5066] Support flink hoodie source metaclient cache
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7017:
URL: https://github.com/apache/hudi/pull/7017#issuecomment-1305050913
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12418",
"triggerID" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"triggerType" : "PUSH"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290274555",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "0000",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1290258456",
"triggerType" : "MANUAL"
}, {
"hash" : "bef3d62c19bac652daf1c3cd351adb06503a1d17",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12565",
"triggerID" : "1291517242",
"triggerType" : "MANUAL"
}, {
"hash" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12792",
"triggerID" : "78c0d932bdc06ab1b28d408d3b04d0a37bc160a7",
"triggerType" : "PUSH"
}, {
"hash" : "836ce4252e35c00d246ea4e1097d4ce5bdfc4a91",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12823",
"triggerID" : "836ce4252e35c00d246ea4e1097d4ce5bdfc4a91",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 78c0d932bdc06ab1b28d408d3b04d0a37bc160a7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12792)
* 836ce4252e35c00d246ea4e1097d4ce5bdfc4a91 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12823)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org