You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "codope (via GitHub)" <gi...@apache.org> on 2024/10/28 18:08:20 UTC
[PR] [MINOR] Init partition stats based on latest file slices only [hudi]
codope opened a new pull request, #12174:
URL: https://github.com/apache/hudi/pull/12174
### Change Logs
Initialize partition stats based on latest file slices only instead of all files in a partition.
### Impact
Initalization of partition stats should take much less time.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2470181229
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578)
* 4c7cfbf4525e29c91fd0539c032bf8c363d82b93 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472558323
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
}, {
"hash" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609",
"triggerID" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"triggerType" : "PUSH"
}, {
"hash" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"triggerType" : "PUSH"
}, {
"hash" : "9f8f90f233b3f2b3e2500fa6c683db87acf6f486",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f8f90f233b3f2b3e2500fa6c683db87acf6f486",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6d9713081729b9570dee54fc83685da5ad9c8cc Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609)
* 101326a7aa948401bd9188bca6b3cd9b1b9cde59 UNKNOWN
* 9f8f90f233b3f2b3e2500fa6c683db87acf6f486 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2468863469
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1beff498959fdfb8e8cbedaa56beb2d0ceda801 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441)
* f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2468985513
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1837097511
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -503,8 +503,8 @@ private String generateUniqueInstantTime(String initializationTime) {
}
}
- private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) {
- HoodieData<HoodieRecord> records = HoodieTableMetadataUtil.convertFilesToPartitionStatsRecords(engineContext, partitionInfoList, dataWriteConfig.getMetadataConfig(), dataMetaClient,
+ private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) throws IOException {
+ HoodieData<HoodieRecord> records = HoodieTableMetadataUtil.convertFilesToPartitionStatsRecords(engineContext, getPartitionFileSlicePairs(), dataWriteConfig.getMetadataConfig(), dataMetaClient,
Review Comment:
Is there a way to get the latest file slices including the ones due to current inflight instant? We are not passing HoodieCommitMetadata to initializeFromFileSystem, so I guess we ay have to resort to listing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472546019
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
}, {
"hash" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609",
"triggerID" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"triggerType" : "PUSH"
}, {
"hash" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e34af6a7cdcbfc7d0c615cd86704362d84ea3934 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607)
* c6d9713081729b9570dee54fc83685da5ad9c8cc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609)
* 101326a7aa948401bd9188bca6b3cd9b1b9cde59 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472453069
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
}, {
"hash" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609",
"triggerID" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e34af6a7cdcbfc7d0c615cd86704362d84ea3934 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607)
* c6d9713081729b9570dee54fc83685da5ad9c8cc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2470322769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c7cfbf4525e29c91fd0539c032bf8c363d82b93 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472265842
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c7cfbf4525e29c91fd0539c032bf8c363d82b93 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590)
* 9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2470177362
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578)
* 4c7cfbf4525e29c91fd0539c032bf8c363d82b93 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472369018
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603)
* e34af6a7cdcbfc7d0c615cd86704362d84ea3934 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2442289867
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1beff498959fdfb8e8cbedaa56beb2d0ceda801 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1838690008
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -207,7 +207,7 @@ private void initMetadataReader() {
}
private HoodieMetadataFileSystemView getMetadataView() {
- if (metadataView == null) {
+ if (metadataView == null || !metadataView.equals(metadata.getMetadataFileSystemView())) {
Review Comment:
sure. we understand the issue. myself and sagar are aligned on the fix here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1838356044
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -207,7 +207,7 @@ private void initMetadataReader() {
}
private HoodieMetadataFileSystemView getMetadataView() {
- if (metadataView == null) {
+ if (metadataView == null || !metadataView.equals(metadata.getMetadataFileSystemView())) {
Review Comment:
Without much context: lets not do any fancy optimizations yet or handle in vs out of sync
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472422778
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
}, {
"hash" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603)
* e34af6a7cdcbfc7d0c615cd86704362d84ea3934 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607)
* c6d9713081729b9570dee54fc83685da5ad9c8cc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2468860979
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1beff498959fdfb8e8cbedaa56beb2d0ceda801 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441)
* f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1837871766
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -503,8 +503,8 @@ private String generateUniqueInstantTime(String initializationTime) {
}
}
- private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) {
- HoodieData<HoodieRecord> records = HoodieTableMetadataUtil.convertFilesToPartitionStatsRecords(engineContext, partitionInfoList, dataWriteConfig.getMetadataConfig(), dataMetaClient,
+ private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) throws IOException {
+ HoodieData<HoodieRecord> records = HoodieTableMetadataUtil.convertFilesToPartitionStatsRecords(engineContext, getPartitionFileSlicePairs(), dataWriteConfig.getMetadataConfig(), dataMetaClient,
Review Comment:
never mind.. i figured it out. This is not what I needed. There was another issue where fs view was not in sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2442464095
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1beff498959fdfb8e8cbedaa56beb2d0ceda801 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472267405
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4c7cfbf4525e29c91fd0539c032bf8c363d82b93 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590)
* 9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1839338784
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -503,8 +493,8 @@ private String generateUniqueInstantTime(String initializationTime) {
}
}
- private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) {
- HoodieData<HoodieRecord> records = HoodieTableMetadataUtil.convertFilesToPartitionStatsRecords(engineContext, partitionInfoList, dataWriteConfig.getMetadataConfig(), dataMetaClient,
+ private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) throws IOException {
Review Comment:
Hmm... any reason why was it being initialized in that method before? https://github.com/apache/hudi/blob/7530e4fa48fb6c32e9cafb587914521bbbb4bc23/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java#L1671
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1839341360
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -503,8 +493,8 @@ private String generateUniqueInstantTime(String initializationTime) {
}
}
- private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) {
- HoodieData<HoodieRecord> records = HoodieTableMetadataUtil.convertFilesToPartitionStatsRecords(engineContext, partitionInfoList, dataWriteConfig.getMetadataConfig(), dataMetaClient,
+ private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) throws IOException {
Review Comment:
not sure. probably we missed to notice that there is an existing MDT FSV that we can use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #12174:
URL: https://github.com/apache/hudi/pull/12174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Init partition stats based on latest file slices only [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1823436083
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2136,12 +2136,44 @@ public static HoodieData<HoodieRecord> convertFilesToPartitionStatsRecords(Hoodi
return engineContext.emptyHoodieData();
}
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndex);
+
+ // Group by partition path and collect file names (BaseFile and LogFiles)
+ Map<String, List<String>> partitionToFileNamesMap = partitionInfoList.stream()
+ .collect(Collectors.toMap(
+ Pair::getKey, // Group by partition path (key of the Pair)
+ pair -> {
+ // Get the FileSlice from the pair
+ FileSlice fileSlice = pair.getValue();
+
+ // Collect BaseFile name if present
+ List<String> fileNames = new ArrayList<>();
+ fileSlice.getBaseFile().ifPresent(baseFile -> fileNames.add(baseFile.getFileName()));
+
+ // Collect LogFile names if present
+ fileSlice.getLogFiles()
+ .map(HoodieLogFile::getFileName)
+ .forEach(fileNames::add);
+
+ return fileNames;
+ },
+ // In case of duplicate keys, merge lists of file names
+ (existingList, newList) -> {
+ existingList.addAll(newList);
+ return existingList;
+ }
+ ));
+
+ // Convert the Map<String, List<String>> to List<Pair<String, List<String>>>
+ List<Pair<String, List<String>>> partitionFileNamePairs = partitionToFileNamesMap.entrySet().stream()
Review Comment:
why do we need to prepare partitionToFileNamesMap and then partitionFileNamePairs.
why can't we directly do flatMap for every partitionInfoList and get List<Pair<String, List<String>>> in a single call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472605958
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
}, {
"hash" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609",
"triggerID" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"triggerType" : "PUSH"
}, {
"hash" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"triggerType" : "PUSH"
}, {
"hash" : "9f8f90f233b3f2b3e2500fa6c683db87acf6f486",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1610",
"triggerID" : "9f8f90f233b3f2b3e2500fa6c683db87acf6f486",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6d9713081729b9570dee54fc83685da5ad9c8cc Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609)
* 101326a7aa948401bd9188bca6b3cd9b1b9cde59 UNKNOWN
* 9f8f90f233b3f2b3e2500fa6c683db87acf6f486 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1610)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472779855
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
}, {
"hash" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609",
"triggerID" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"triggerType" : "PUSH"
}, {
"hash" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"triggerType" : "PUSH"
}, {
"hash" : "9f8f90f233b3f2b3e2500fa6c683db87acf6f486",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1610",
"triggerID" : "9f8f90f233b3f2b3e2500fa6c683db87acf6f486",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 101326a7aa948401bd9188bca6b3cd9b1b9cde59 UNKNOWN
* 9f8f90f233b3f2b3e2500fa6c683db87acf6f486 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1610)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1838342292
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -207,7 +207,7 @@ private void initMetadataReader() {
}
private HoodieMetadataFileSystemView getMetadataView() {
- if (metadataView == null) {
+ if (metadataView == null || !metadataView.equals(metadata.getMetadataFileSystemView())) {
Review Comment:
I am not sure I get this fix. our Metadata writers are shotlived in general. We just create a new writer, initial new partitions, apply updates for a given commit and close it out.
So, how come metadataView can be out of sync ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1839321040
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -503,8 +493,8 @@ private String generateUniqueInstantTime(String initializationTime) {
}
}
- private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) {
- HoodieData<HoodieRecord> records = HoodieTableMetadataUtil.convertFilesToPartitionStatsRecords(engineContext, partitionInfoList, dataWriteConfig.getMetadataConfig(), dataMetaClient,
+ private Pair<Integer, HoodieData<HoodieRecord>> initializePartitionStatsIndex(List<DirectoryInfo> partitionInfoList) throws IOException {
Review Comment:
I am ok to initialize new FSV while we are trying to initialize a new MDT partition.
but lets not initialize new MDT FSV for regular updates.
for eg, getRecordIndexReplacedRecords() should not be initializing a new MDT fsv and just re-use existing one.
wdyt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [MINOR] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2442292887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b1beff498959fdfb8e8cbedaa56beb2d0ceda801 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472367449
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603)
* e34af6a7cdcbfc7d0c615cd86704362d84ea3934 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472348134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1837079699
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2136,12 +2136,44 @@ public static HoodieData<HoodieRecord> convertFilesToPartitionStatsRecords(Hoodi
return engineContext.emptyHoodieData();
}
LOG.debug("Indexing following columns for partition stats index: {}", columnsToIndex);
+
+ // Group by partition path and collect file names (BaseFile and LogFiles)
+ Map<String, List<String>> partitionToFileNamesMap = partitionInfoList.stream()
+ .collect(Collectors.toMap(
+ Pair::getKey, // Group by partition path (key of the Pair)
+ pair -> {
+ // Get the FileSlice from the pair
+ FileSlice fileSlice = pair.getValue();
+
+ // Collect BaseFile name if present
+ List<String> fileNames = new ArrayList<>();
+ fileSlice.getBaseFile().ifPresent(baseFile -> fileNames.add(baseFile.getFileName()));
+
+ // Collect LogFile names if present
+ fileSlice.getLogFiles()
+ .map(HoodieLogFile::getFileName)
+ .forEach(fileNames::add);
+
+ return fileNames;
+ },
+ // In case of duplicate keys, merge lists of file names
+ (existingList, newList) -> {
+ existingList.addAll(newList);
+ return existingList;
+ }
+ ));
+
+ // Convert the Map<String, List<String>> to List<Pair<String, List<String>>>
+ List<Pair<String, List<String>>> partitionFileNamePairs = partitionToFileNamesMap.entrySet().stream()
Review Comment:
both engineContext.parallelize and flatMap work with list data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472424970
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
}, {
"hash" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e34af6a7cdcbfc7d0c615cd86704362d84ea3934 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607)
* c6d9713081729b9570dee54fc83685da5ad9c8cc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12174:
URL: https://github.com/apache/hudi/pull/12174#issuecomment-2472551661
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1441",
"triggerID" : "b1beff498959fdfb8e8cbedaa56beb2d0ceda801",
"triggerType" : "PUSH"
}, {
"hash" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1578",
"triggerID" : "f3cb48a03e4bd5261bbd794a8dbfa30855ea9f09",
"triggerType" : "PUSH"
}, {
"hash" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1590",
"triggerID" : "4c7cfbf4525e29c91fd0539c032bf8c363d82b93",
"triggerType" : "PUSH"
}, {
"hash" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1603",
"triggerID" : "9ce98f0cef31d6e93e12a2b3b7d7b8b4e207911a",
"triggerType" : "PUSH"
}, {
"hash" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1607",
"triggerID" : "e34af6a7cdcbfc7d0c615cd86704362d84ea3934",
"triggerType" : "PUSH"
}, {
"hash" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609",
"triggerID" : "c6d9713081729b9570dee54fc83685da5ad9c8cc",
"triggerType" : "PUSH"
}, {
"hash" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "101326a7aa948401bd9188bca6b3cd9b1b9cde59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6d9713081729b9570dee54fc83685da5ad9c8cc Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1609)
* 101326a7aa948401bd9188bca6b3cd9b1b9cde59 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8452] Init partition stats based on latest file slices only [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #12174:
URL: https://github.com/apache/hudi/pull/12174#discussion_r1839487561
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java:
##########
@@ -360,7 +360,7 @@ public final class HoodieMetadataConfig extends HoodieConfig {
public static final ConfigProperty<Boolean> PARTITION_STATS_INDEX_CONSOLIDATE_ON_EVERY_WRITE = ConfigProperty
.key(METADATA_PREFIX + ".index.partition.stats.consolidate.on.every.write")
- .defaultValue(false)
+ .defaultValue(true)
Review Comment:
So, we could remove this config totally as well right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org