You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2023/07/05 02:07:21 UTC
[GitHub] [hudi] Zouxxyy opened a new pull request, #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Zouxxyy opened a new pull request, #9122:
URL: https://github.com/apache/hudi/pull/9122
… HoodieFileIndex
### Change Logs
Currently there is a lazy list mechanism in `hoodieFileIndex`, but it only takes effect during initialization. We can make it take effect when refresh. At present, almost all spark commands in HUDI will do refresh, such as DDL alter table operation, we don’t need to do list file at all
### Impact
Lazy fetching partition path & file slice when refresh in HoodieFileIndex
### Risk level (write none, low medium or high below)
medium
### Documentation Update
none
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9122:
URL: https://github.com/apache/hudi/pull/9122#discussion_r1252517112
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -144,18 +144,7 @@ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
this.engineContext = engineContext;
this.fileStatusCache = fileStatusCache;
- // The `shouldListLazily` variable controls how we initialize the TableFileIndex:
- // - non-lazy/eager listing (shouldListLazily=false): all partitions and file slices will be loaded eagerly during initialization.
- // - lazy listing (shouldListLazily=true): partitions listing will be done lazily with the knowledge from query predicate on partition
- // columns. And file slices fetching only happens for partitions satisfying the given filter.
- //
- // In SparkSQL, `shouldListLazily` is controlled by option `REFRESH_PARTITION_AND_FILES_IN_INITIALIZATION`.
- // In lazy listing case, if no predicate on partition is provided, all partitions will still be loaded.
- if (shouldListLazily) {
- this.tableMetadata = createMetadataTable(engineContext, metadataConfig, basePath);
Review Comment:
The initialization of `tableMetadata` is removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9122:
URL: https://github.com/apache/hudi/pull/9122#issuecomment-1621585706
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18324",
"triggerID" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"triggerType" : "PUSH"
}, {
"hash" : "fa8a7ec1cdf1d90344fc9c24fa6a278aedf5e44f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18332",
"triggerID" : "fa8a7ec1cdf1d90344fc9c24fa6a278aedf5e44f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa8a7ec1cdf1d90344fc9c24fa6a278aedf5e44f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18332)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #9122:
URL: https://github.com/apache/hudi/pull/9122
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9122:
URL: https://github.com/apache/hudi/pull/9122#discussion_r1252517410
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -144,18 +144,7 @@ public BaseHoodieTableFileIndex(HoodieEngineContext engineContext,
this.engineContext = engineContext;
this.fileStatusCache = fileStatusCache;
- // The `shouldListLazily` variable controls how we initialize the TableFileIndex:
- // - non-lazy/eager listing (shouldListLazily=false): all partitions and file slices will be loaded eagerly during initialization.
- // - lazy listing (shouldListLazily=true): partitions listing will be done lazily with the knowledge from query predicate on partition
- // columns. And file slices fetching only happens for partitions satisfying the given filter.
- //
- // In SparkSQL, `shouldListLazily` is controlled by option `REFRESH_PARTITION_AND_FILES_IN_INITIALIZATION`.
- // In lazy listing case, if no predicate on partition is provided, all partitions will still be loaded.
- if (shouldListLazily) {
- this.tableMetadata = createMetadataTable(engineContext, metadataConfig, basePath);
Review Comment:
Ignore, it is created in `doRefresh`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9122:
URL: https://github.com/apache/hudi/pull/9122#issuecomment-1621135940
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18324",
"triggerID" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"triggerType" : "PUSH"
}, {
"hash" : "fa8a7ec1cdf1d90344fc9c24fa6a278aedf5e44f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa8a7ec1cdf1d90344fc9c24fa6a278aedf5e44f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f44e3cd97dbc108faabcdd5da0d805b1680e211 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18324)
* fa8a7ec1cdf1d90344fc9c24fa6a278aedf5e44f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9122:
URL: https://github.com/apache/hudi/pull/9122#issuecomment-1621023436
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18324",
"triggerID" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f44e3cd97dbc108faabcdd5da0d805b1680e211 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9122:
URL: https://github.com/apache/hudi/pull/9122#issuecomment-1620937689
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f44e3cd97dbc108faabcdd5da0d805b1680e211 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9122:
URL: https://github.com/apache/hudi/pull/9122#issuecomment-1621204348
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18324",
"triggerID" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"triggerType" : "PUSH"
}, {
"hash" : "fa8a7ec1cdf1d90344fc9c24fa6a278aedf5e44f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18332",
"triggerID" : "fa8a7ec1cdf1d90344fc9c24fa6a278aedf5e44f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f44e3cd97dbc108faabcdd5da0d805b1680e211 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18324)
* fa8a7ec1cdf1d90344fc9c24fa6a278aedf5e44f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18332)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9122: [HUDI-6477] Lazy fetching partition path & file slice when refresh in…
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9122:
URL: https://github.com/apache/hudi/pull/9122#issuecomment-1620942995
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18324",
"triggerID" : "2f44e3cd97dbc108faabcdd5da0d805b1680e211",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f44e3cd97dbc108faabcdd5da0d805b1680e211 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org