You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2025/10/03 02:06:03 UTC
[PR] Perf mdt expression index [hudi]
jonvex opened a new pull request, #14048:
URL: https://github.com/apache/hudi/pull/14048
### Describe the issue this Pull Request addresses
<!-- Either describe the issue inline here with motivation behind the changes
(or) link to an issue by including `Closes #<issue-number>` for context.
If this PR includes changes to the storage format, public APIs,
or has breaking changes, use `!` (e.g., feat!: ...) -->
### Summary and Changelog
<!-- Short, plain-English summary of what users gain or what changed in behavior.
Followed by a detailed log of all the changes. Highlight if any code was copied. -->
### Impact
<!-- Describe any public API or user-facing feature change or any performance impact. -->
### Risk Level
<!-- Accepted values: none, low, medium or high. Other than `none`, explain the risk.
If medium or high, explain what verification was done to mitigate the risks. -->
### Documentation Update
<!-- Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
- The config description must be updated if new configs are added or the default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website. Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website. -->
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Enough context is provided in the sections above
- [ ] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3363999190
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63af2290e014250070f73990fe5c4cda196c77da Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3367578576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
}, {
"hash" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078",
"triggerID" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"triggerType" : "PUSH"
}, {
"hash" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079",
"triggerID" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"triggerType" : "PUSH"
}, {
"hash" : "a209950853212f0f6d1bfd70a7c06e893e850170",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9084",
"triggerID" : "a209950853212f0f6d1bfd70a7c06e893e850170",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a209950853212f0f6d1bfd70a7c06e893e850170 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9084)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3366861105
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9c72e5e86ab04ea66e5f4e1e2f192056b234691 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061)
* 4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073)
* d94249098bebc9c6aca1329ae46b869ce1a30a4e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3367162307
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
}, {
"hash" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078",
"triggerID" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"triggerType" : "PUSH"
}, {
"hash" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079",
"triggerID" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36511af05242e2251f216841b6a56a8aa78b40a3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3363890696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63af2290e014250070f73990fe5c4cda196c77da Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3366879745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
}, {
"hash" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073)
* d94249098bebc9c6aca1329ae46b869ce1a30a4e Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076)
* 8882623a9f7795ce9b567533bab35cbfa26c894b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #14048:
URL: https://github.com/apache/hudi/pull/14048
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3365254465
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9c72e5e86ab04ea66e5f4e1e2f192056b234691 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3366900283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
}, {
"hash" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078",
"triggerID" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"triggerType" : "PUSH"
}, {
"hash" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073)
* 8882623a9f7795ce9b567533bab35cbfa26c894b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078)
* 36511af05242e2251f216841b6a56a8aa78b40a3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3366872157
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
}, {
"hash" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9c72e5e86ab04ea66e5f4e1e2f192056b234691 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061)
* 4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073)
* d94249098bebc9c6aca1329ae46b869ce1a30a4e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076)
* 8882623a9f7795ce9b567533bab35cbfa26c894b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3363888266
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63af2290e014250070f73990fe5c4cda196c77da UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing and incorrect stats for expression index [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14048:
URL: https://github.com/apache/hudi/pull/14048#discussion_r2403029439
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -406,22 +408,25 @@ public static HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>
// Step 2: Compute expression index records for the modified partitions
LOG.debug("Indexing following columns for partition stats index: {}", validColumnsToIndex);
- List<String> partitionPaths = new ArrayList<>(commitMetadata.getWritePartitionPaths());
- HoodieTableFileSystemView fileSystemView = getFileSystemViewForMetadataTable(dataMetaClient);
- int parallelism = Math.max(Math.min(partitionPaths.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
- return engineContext.parallelize(partitionPaths, parallelism).mapToPair(partitionName -> {
+ List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(commitMetadata.getWriteStats().stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
+ .values());
+
+ Map<String, Set<String>> fileGroupIdsToReplaceMap = (commitMetadata instanceof HoodieReplaceCommitMetadata)
+ ? ((HoodieReplaceCommitMetadata) commitMetadata).getPartitionToReplaceFileIds()
+ .entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, e -> new HashSet<>(e.getValue())))
+ : Collections.emptyMap();
+
+ String maxInstantTime = HoodieMetadataWriteUtils.getMaxInstantTime(dataMetaClient, instantTime);
+ int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
+ return engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
+ final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
// Collect Column Metadata for Each File part of active file system view of latest snapshot
// Get all file names, including log files, in a set from the file slices
- // TODO(yihua): fix this
- Set<String> fileNames = HoodieTableMetadataUtil.getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.of(fileSystemView), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
+ Set<String> fileNames = HoodieMetadataWriteUtils.getFilesToFetchColumnStats(partitionedWriteStat, dataMetaClient, tableMetadata, dataWriteConfig, partitionName, maxInstantTime, instantTime,
Review Comment:
Got it, that makes sense. I renamed the method to avoid confusion and make it clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3367439974
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
}, {
"hash" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078",
"triggerID" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"triggerType" : "PUSH"
}, {
"hash" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079",
"triggerID" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"triggerType" : "PUSH"
}, {
"hash" : "a209950853212f0f6d1bfd70a7c06e893e850170",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9084",
"triggerID" : "a209950853212f0f6d1bfd70a7c06e893e850170",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36511af05242e2251f216841b6a56a8aa78b40a3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079)
* a209950853212f0f6d1bfd70a7c06e893e850170 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9084)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3366643262
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9c72e5e86ab04ea66e5f4e1e2f192056b234691 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061)
* 4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3366635875
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9c72e5e86ab04ea66e5f4e1e2f192056b234691 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061)
* 4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3366926306
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
}, {
"hash" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078",
"triggerID" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"triggerType" : "PUSH"
}, {
"hash" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079",
"triggerID" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8882623a9f7795ce9b567533bab35cbfa26c894b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078)
* 36511af05242e2251f216841b6a56a8aa78b40a3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14048:
URL: https://github.com/apache/hudi/pull/14048#discussion_r2400603555
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java:
##########
@@ -542,4 +512,51 @@ private static StoragePathInfo getLogFileStoragePathInfo(HoodieLogFile logFile)
}
return new StoragePathInfo(logFile.getPath(), logFile.getFileSize(), false, (short) 0, 0, 0);
}
+
+ public static String getMaxInstantTime(HoodieTableMetaClient dataMetaClient, String instantTime) {
+ Option<String> lastCompletedInstant = dataMetaClient.getActiveTimeline().filterCompletedInstants()
+ .lastInstant()
+ .map(HoodieInstant::requestedTime);
+ return lastCompletedInstant.map(lastCompletedInstantTime ->
+ lastCompletedInstantTime.compareTo(instantTime) > 0 ? lastCompletedInstantTime : instantTime).orElse(instantTime);
+ }
+
+ /**
+ * Collect column metadata of each file that does not have column stats provided by the write stat in the commit metadata
+ */
+ public static Set<String> getFilesToFetchColumnStats(List<HoodieWriteStat> partitionedWriteStat,
+ HoodieTableMetaClient dataMetaClient,
+ HoodieTableMetadata tableMetadata,
+ HoodieWriteConfig dataWriteConfig,
+ String partitionName,
+ String maxInstantTime,
+ String instantTime) {
+ // Get the latest merged file slices based on the commited files part of the latest snapshot and the new files of the current commit metadata
+ // Get the latest merged file slices based on the commited files part of the latest snapshot and the new files of the current commit metadata
+ List<StoragePathInfo> consolidatedPathInfos = new ArrayList<>();
+ partitionedWriteStat.forEach(
+ stat -> consolidatedPathInfos.add(
+ new StoragePathInfo(new StoragePath(dataMetaClient.getBasePath(), stat.getPath()), stat.getFileSizeInBytes(), false, (short) 0, 0, 0)));
+ SyncableFileSystemView fileSystemViewForCommitedFiles = FileSystemViewManager.createViewManager(new HoodieLocalEngineContext(dataMetaClient.getStorageConf()),
+ dataWriteConfig.getMetadataConfig(), dataWriteConfig.getViewStorageConfig(), dataWriteConfig.getCommonConfig(),
Review Comment:
guess the clustering fix that ethan is yet to add to partition stats index, also need to be applied here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #14048:
URL: https://github.com/apache/hudi/pull/14048#discussion_r2402855471
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -406,22 +408,25 @@ public static HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>
// Step 2: Compute expression index records for the modified partitions
LOG.debug("Indexing following columns for partition stats index: {}", validColumnsToIndex);
- List<String> partitionPaths = new ArrayList<>(commitMetadata.getWritePartitionPaths());
- HoodieTableFileSystemView fileSystemView = getFileSystemViewForMetadataTable(dataMetaClient);
- int parallelism = Math.max(Math.min(partitionPaths.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
- return engineContext.parallelize(partitionPaths, parallelism).mapToPair(partitionName -> {
+ List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(commitMetadata.getWriteStats().stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
+ .values());
+
+ Map<String, Set<String>> fileGroupIdsToReplaceMap = (commitMetadata instanceof HoodieReplaceCommitMetadata)
+ ? ((HoodieReplaceCommitMetadata) commitMetadata).getPartitionToReplaceFileIds()
+ .entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, e -> new HashSet<>(e.getValue())))
+ : Collections.emptyMap();
+
+ String maxInstantTime = HoodieMetadataWriteUtils.getMaxInstantTime(dataMetaClient, instantTime);
+ int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
+ return engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
+ final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
// Collect Column Metadata for Each File part of active file system view of latest snapshot
// Get all file names, including log files, in a set from the file slices
- // TODO(yihua): fix this
- Set<String> fileNames = HoodieTableMetadataUtil.getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.of(fileSystemView), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
+ Set<String> fileNames = HoodieMetadataWriteUtils.getFilesToFetchColumnStats(partitionedWriteStat, dataMetaClient, tableMetadata, dataWriteConfig, partitionName, maxInstantTime, instantTime,
Review Comment:
For expression index, we have to compute the stats for all files based on the expressions including the files in the `partitionedWriteStat`, correct?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataWriteUtils.java:
##########
@@ -471,44 +471,18 @@ public static HoodieData<HoodieRecord> convertMetadataToPartitionStatRecords(Hoo
Option<String> lastCompletedInstant = dataMetaClient.getActiveTimeline().filterCompletedInstants()
Review Comment:
This can be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3364949730
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63af2290e014250070f73990fe5c4cda196c77da Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049)
* f9c72e5e86ab04ea66e5f4e1e2f192056b234691 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing and incorrect stats for expression index [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #14048:
URL: https://github.com/apache/hudi/pull/14048#discussion_r2402899933
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -406,22 +408,25 @@ public static HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>
// Step 2: Compute expression index records for the modified partitions
LOG.debug("Indexing following columns for partition stats index: {}", validColumnsToIndex);
- List<String> partitionPaths = new ArrayList<>(commitMetadata.getWritePartitionPaths());
- HoodieTableFileSystemView fileSystemView = getFileSystemViewForMetadataTable(dataMetaClient);
- int parallelism = Math.max(Math.min(partitionPaths.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
- return engineContext.parallelize(partitionPaths, parallelism).mapToPair(partitionName -> {
+ List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(commitMetadata.getWriteStats().stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
+ .values());
+
+ Map<String, Set<String>> fileGroupIdsToReplaceMap = (commitMetadata instanceof HoodieReplaceCommitMetadata)
+ ? ((HoodieReplaceCommitMetadata) commitMetadata).getPartitionToReplaceFileIds()
+ .entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, e -> new HashSet<>(e.getValue())))
+ : Collections.emptyMap();
+
+ String maxInstantTime = HoodieMetadataWriteUtils.getMaxInstantTime(dataMetaClient, instantTime);
+ int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
+ return engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
+ final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
// Collect Column Metadata for Each File part of active file system view of latest snapshot
// Get all file names, including log files, in a set from the file slices
- // TODO(yihua): fix this
- Set<String> fileNames = HoodieTableMetadataUtil.getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.of(fileSystemView), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
+ Set<String> fileNames = HoodieMetadataWriteUtils.getFilesToFetchColumnStats(partitionedWriteStat, dataMetaClient, tableMetadata, dataWriteConfig, partitionName, maxInstantTime, instantTime,
Review Comment:
expr index is slightly different.
can you go back to the caller of this.
bcoz, we can't compute the expr index from within spark task, expr index compuation is slightly different from partition stats index , where we compute the col stats for individual files within the same spark task.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/utils/SparkMetadataWriterUtils.java:
##########
@@ -406,22 +408,25 @@ public static HoodiePairData<String, List<HoodieColumnRangeMetadata<Comparable>>
// Step 2: Compute expression index records for the modified partitions
LOG.debug("Indexing following columns for partition stats index: {}", validColumnsToIndex);
- List<String> partitionPaths = new ArrayList<>(commitMetadata.getWritePartitionPaths());
- HoodieTableFileSystemView fileSystemView = getFileSystemViewForMetadataTable(dataMetaClient);
- int parallelism = Math.max(Math.min(partitionPaths.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
- return engineContext.parallelize(partitionPaths, parallelism).mapToPair(partitionName -> {
+ List<List<HoodieWriteStat>> partitionedWriteStats = new ArrayList<>(commitMetadata.getWriteStats().stream()
+ .collect(Collectors.groupingBy(HoodieWriteStat::getPartitionPath))
+ .values());
+
+ Map<String, Set<String>> fileGroupIdsToReplaceMap = (commitMetadata instanceof HoodieReplaceCommitMetadata)
+ ? ((HoodieReplaceCommitMetadata) commitMetadata).getPartitionToReplaceFileIds()
+ .entrySet().stream().collect(Collectors.toMap(Map.Entry::getKey, e -> new HashSet<>(e.getValue())))
+ : Collections.emptyMap();
+
+ String maxInstantTime = HoodieMetadataWriteUtils.getMaxInstantTime(dataMetaClient, instantTime);
+ int parallelism = Math.max(Math.min(partitionedWriteStats.size(), metadataConfig.getPartitionStatsIndexParallelism()), 1);
+ return engineContext.parallelize(partitionedWriteStats, parallelism).mapToPair(partitionedWriteStat -> {
+ final String partitionName = partitionedWriteStat.get(0).getPartitionPath();
checkState(tableMetadata != null, "tableMetadata should not be null when scanning metadata table");
// Collect Column Metadata for Each File part of active file system view of latest snapshot
// Get all file names, including log files, in a set from the file slices
- // TODO(yihua): fix this
- Set<String> fileNames = HoodieTableMetadataUtil.getPartitionLatestFileSlicesIncludingInflight(dataMetaClient, Option.of(fileSystemView), partitionName).stream()
- .flatMap(fileSlice -> Stream.concat(
- Stream.of(fileSlice.getBaseFile().map(HoodieBaseFile::getFileName).orElse(null)),
- fileSlice.getLogFiles().map(HoodieLogFile::getFileName)))
- .filter(Objects::nonNull)
- .collect(Collectors.toSet());
+ Set<String> fileNames = HoodieMetadataWriteUtils.getFilesToFetchColumnStats(partitionedWriteStat, dataMetaClient, tableMetadata, dataWriteConfig, partitionName, maxInstantTime, instantTime,
Review Comment:
expr index is slightly different.
can you go back to the caller of this.
bcoz, we can't compute the expr index from within spark task, expr index compuation is slightly different from partition stats index , where we compute the col stats for individual files within a map call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3367436007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
}, {
"hash" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078",
"triggerID" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"triggerType" : "PUSH"
}, {
"hash" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079",
"triggerID" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"triggerType" : "PUSH"
}, {
"hash" : "a209950853212f0f6d1bfd70a7c06e893e850170",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a209950853212f0f6d1bfd70a7c06e893e850170",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 36511af05242e2251f216841b6a56a8aa78b40a3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079)
* a209950853212f0f6d1bfd70a7c06e893e850170 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Perf mdt expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3364945173
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63af2290e014250070f73990fe5c4cda196c77da Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049)
* f9c72e5e86ab04ea66e5f4e1e2f192056b234691 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3366904830
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9076",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
}, {
"hash" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078",
"triggerID" : "8882623a9f7795ce9b567533bab35cbfa26c894b",
"triggerType" : "PUSH"
}, {
"hash" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079",
"triggerID" : "36511af05242e2251f216841b6a56a8aa78b40a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073)
* 8882623a9f7795ce9b567533bab35cbfa26c894b Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9078)
* 36511af05242e2251f216841b6a56a8aa78b40a3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9079)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Fix record generation to avoid file system based listing and incorrect stats for expression index [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #14048:
URL: https://github.com/apache/hudi/pull/14048#issuecomment-3366857465
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63af2290e014250070f73990fe5c4cda196c77da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9049",
"triggerID" : "63af2290e014250070f73990fe5c4cda196c77da",
"triggerType" : "PUSH"
}, {
"hash" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061",
"triggerID" : "f9c72e5e86ab04ea66e5f4e1e2f192056b234691",
"triggerType" : "PUSH"
}, {
"hash" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073",
"triggerID" : "4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b",
"triggerType" : "PUSH"
}, {
"hash" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d94249098bebc9c6aca1329ae46b869ce1a30a4e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9c72e5e86ab04ea66e5f4e1e2f192056b234691 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9061)
* 4f345134e9ef83e7f38c25ac1c2b8b183f6d7e3b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=9073)
* d94249098bebc9c6aca1329ae46b869ce1a30a4e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org