You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2025/04/02 05:22:00 UTC
[PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
TheR1sing3un opened a new pull request, #13070:
URL: https://github.com/apache/hudi/pull/13070
close to: https://github.com/apache/hudi/issues/12139
When dealing with `HadoopFsRelation`, Spark merges `PartitionedFile` based on data such as file size. At present, we directly use the base file or a random log file as the `PartitionedFile` of the `FileSlice`. As a result, spark cannot accurately use representative data when merging. Therefore, I estimated the size of the entire `FileSlice` file if it is converted into parqeut file. Using this data to represent the file slice can provide more accurate data for spark to optimize.
### Change Logs
1. Introduce a representative file containing the estimated total size of file slice
### Impact
Reduces task tilt when reading
### Risk level (write none, low medium or high below)
low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2821521188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
}, {
"hash" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059",
"triggerID" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"triggerType" : "PUSH"
}, {
"hash" : "1375df06346952de4af3d20daf22b3525d48723f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1375df06346952de4af3d20daf22b3525d48723f",
"triggerType" : "PUSH"
}, {
"hash" : "66a4d7039df6b3d709019fc1699e98f6a5c70c3f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5068",
"triggerID" : "66a4d7039df6b3d709019fc1699e98f6a5c70c3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 1375df06346952de4af3d20daf22b3525d48723f UNKNOWN
* 66a4d7039df6b3d709019fc1699e98f6a5c70c3f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5068)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2820923154
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
}, {
"hash" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059",
"triggerID" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 3c414f6e97dd1184138acad1e0ea96a63e3f99b7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2783063412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 9372b4ba21c1286f378122a6f7975a620c5adca3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2783266143
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 9372b4ba21c1286f378122a6f7975a620c5adca3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2782391286
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* ad42cecbff2da69e2ce4bb0f17f9246d62709bb6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596)
* 9372b4ba21c1286f378122a6f7975a620c5adca3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2817741551
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 3c1fdea4346fadd861101a4c3d1709fd1909734c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2817988509
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 3c1fdea4346fadd861101a4c3d1709fd1909734c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006)
* 025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818473239
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818509889
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771427521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1db2dd895de991a9285a33bfff9c3d7cab7a26 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575)
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 33ee1f487c6a97ace1c82caa6cf4019ec899b568 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771377484
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1db2dd895de991a9285a33bfff9c3d7cab7a26 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575)
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2772301543
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* e522af2c53e6db57bcd88e4249f86e01fb12a8e2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2791478987
@danny0405 @yihua Ready for review again~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2051923527
##########
hudi-common/src/main/java/org/apache/hudi/common/model/FileSlice.java:
##########
@@ -186,4 +186,26 @@ public boolean equals(Object o) {
public int hashCode() {
return Objects.hash(fileGroupId, baseInstantTime);
}
+
+ /**
+ * Get the total file size of a file slice similar on the base file.
+ * For the log file, we need to convert its size to the estimated size similar on the base file in a certain proportion
+ */
+ public long getTotalFileSizeSimilarOnBaseFile(double logFileFraction) {
+ long logFileSize = convertLogFilesSizeSimilarOnBaseFile(logFileFraction);
+ return getBaseFile().isPresent() ? getBaseFile().get().getFileSize() + logFileSize : logFileSize;
+ }
+
+ private long convertLogFilesSizeSimilarOnBaseFile(double logFileFraction) {
Review Comment:
> still think the original method name is better.
done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2783062162
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2052192615
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkPartitionFileUtils.scala:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.BaseHoodieTableFileIndex.PartitionPath
+import org.apache.hudi.common.config.HoodieStorageConfig
+import org.apache.hudi.common.model.FileSlice
+
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources.PartitionDirectory
+
+object SparkPartitionFileUtils extends SparkAdapterSupport {
Review Comment:
`PartitionDirectoryConverter`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2819883933
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2820095687
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 930b3491aece079a7f63a4fec22ed2a667a0d606 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2820499439
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
}, {
"hash" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 0c9215910b27eb10c946bb72819b8ebe75129ca3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055)
* 3c414f6e97dd1184138acad1e0ea96a63e3f99b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2026278620
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -178,8 +183,8 @@ class HoodieFileGroupReaderBasedParquetFileFormat(tablePath: String,
internalSchemaOpt,
metaClient,
props,
- file.start,
- file.length,
+ 0,
Review Comment:
then why the start is constant 0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2025882382
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -178,8 +183,8 @@ class HoodieFileGroupReaderBasedParquetFileFormat(tablePath: String,
internalSchemaOpt,
metaClient,
props,
- file.start,
- file.length,
+ 0,
Review Comment:
can you elaborate this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818089313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017)
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2821010633
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
}, {
"hash" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059",
"triggerID" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"triggerType" : "PUSH"
}, {
"hash" : "1375df06346952de4af3d20daf22b3525d48723f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1375df06346952de4af3d20daf22b3525d48723f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 3c414f6e97dd1184138acad1e0ea96a63e3f99b7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059)
* 1375df06346952de4af3d20daf22b3525d48723f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2820196104
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 930b3491aece079a7f63a4fec22ed2a667a0d606 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041)
* 0c9215910b27eb10c946bb72819b8ebe75129ca3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2820495888
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
}, {
"hash" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 930b3491aece079a7f63a4fec22ed2a667a0d606 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041)
* 0c9215910b27eb10c946bb72819b8ebe75129ca3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055)
* 3c414f6e97dd1184138acad1e0ea96a63e3f99b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2038599649
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -169,26 +169,29 @@ case class HoodieFileIndex(spark: SparkSession,
val prunedPartitionsAndFilteredFileSlices = filterFileSlices(dataFilters, partitionFilters).map {
case (partitionOpt, fileSlices) =>
if (shouldEmbedFileSlices) {
- val baseFileStatusesAndLogFileOnly: Seq[FileStatus] = fileSlices.map(slice => {
- if (slice.getBaseFile.isPresent) {
+ val logFileEstimationFraction = options.getOrElse(HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.key(),
+ HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.defaultValue()).toDouble
+ // 1. Generate a disguised representative file for each file slice, which spark uses to optimize rdd partition parallelism based on data such as file size
Review Comment:
disguised -> delegate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2783342263
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2030685108
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileSliceUtils.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class FileSliceUtils {
+
+ /**
+ * Get the total file size of a file slice in parquet format.
+ * For the log file, we need to convert its size to the estimated size in the parquet format in a certain proportion
+ */
+ public static long getTotalFileSizeAsParquetFormat(FileSlice fileSlice, double logFileFraction) {
Review Comment:
> Similar on the base file.
done~
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/deltacommit/SparkUpsertDeltaCommitPartitioner.java:
##########
@@ -73,13 +74,13 @@ protected List<SmallFile> getSmallFiles(String partitionPath) {
if (smallFileSlice.getBaseFile().isPresent()) {
HoodieBaseFile baseFile = smallFileSlice.getBaseFile().get();
sf.location = new HoodieRecordLocation(baseFile.getCommitTime(), baseFile.getFileId());
- sf.sizeBytes = getTotalFileSize(smallFileSlice);
+ sf.sizeBytes = FileSliceUtils.getTotalFileSizeAsParquetFormat(smallFileSlice, config.getLogFileToParquetCompressionRatio());
Review Comment:
> Can be moved to class `FileSlice`, and keep the comment of method: `convertLogFilesSizeToExpectedParquetSize`.
done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771486466
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1db2dd895de991a9285a33bfff9c3d7cab7a26 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575)
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 33ee1f487c6a97ace1c82caa6cf4019ec899b568 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579)
* 98f8b5a7964fc63138713c8dd5abdaec35452c50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2821033353
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
}, {
"hash" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059",
"triggerID" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"triggerType" : "PUSH"
}, {
"hash" : "1375df06346952de4af3d20daf22b3525d48723f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1375df06346952de4af3d20daf22b3525d48723f",
"triggerType" : "PUSH"
}, {
"hash" : "66a4d7039df6b3d709019fc1699e98f6a5c70c3f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "66a4d7039df6b3d709019fc1699e98f6a5c70c3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 3c414f6e97dd1184138acad1e0ea96a63e3f99b7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059)
* 1375df06346952de4af3d20daf22b3525d48723f UNKNOWN
* 66a4d7039df6b3d709019fc1699e98f6a5c70c3f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2820324633
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 930b3491aece079a7f63a4fec22ed2a667a0d606 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041)
* 0c9215910b27eb10c946bb72819b8ebe75129ca3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2051923424
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -169,26 +169,29 @@ case class HoodieFileIndex(spark: SparkSession,
val prunedPartitionsAndFilteredFileSlices = filterFileSlices(dataFilters, partitionFilters).map {
case (partitionOpt, fileSlices) =>
if (shouldEmbedFileSlices) {
- val baseFileStatusesAndLogFileOnly: Seq[FileStatus] = fileSlices.map(slice => {
- if (slice.getBaseFile.isPresent) {
+ val logFileEstimationFraction = options.getOrElse(HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.key(),
+ HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.defaultValue()).toDouble
+ // 1. Generate a disguised representative file for each file slice, which spark uses to optimize rdd partition parallelism based on data such as file size
Review Comment:
> disguised -> delegate?
done~
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -169,26 +169,29 @@ case class HoodieFileIndex(spark: SparkSession,
val prunedPartitionsAndFilteredFileSlices = filterFileSlices(dataFilters, partitionFilters).map {
case (partitionOpt, fileSlices) =>
if (shouldEmbedFileSlices) {
- val baseFileStatusesAndLogFileOnly: Seq[FileStatus] = fileSlices.map(slice => {
- if (slice.getBaseFile.isPresent) {
+ val logFileEstimationFraction = options.getOrElse(HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.key(),
+ HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.defaultValue()).toDouble
+ // 1. Generate a disguised representative file for each file slice, which spark uses to optimize rdd partition parallelism based on data such as file size
+ // For file slice only has base file, we directly use the base file size as representative file size
+ // For file slice has log file, we estimate the representative file size based on the log file size and option(base file) size
+ val representFiles = fileSlices.map(slice => {
Review Comment:
> delegateFiles
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2817577727
@danny0405 Ready for review again~ Sorry for replying late.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2774466151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* ad42cecbff2da69e2ce4bb0f17f9246d62709bb6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771572615
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 33ee1f487c6a97ace1c82caa6cf4019ec899b568 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579)
* 98f8b5a7964fc63138713c8dd5abdaec35452c50 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2782813143
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 9372b4ba21c1286f378122a6f7975a620c5adca3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2783345334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 9372b4ba21c1286f378122a6f7975a620c5adca3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2819883849
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2817605147
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 9372b4ba21c1286f378122a6f7975a620c5adca3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685)
* 3c1fdea4346fadd861101a4c3d1709fd1909734c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2820014387
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038)
* 930b3491aece079a7f63a4fec22ed2a667a0d606 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2029415352
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileSliceUtils.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class FileSliceUtils {
+
+ /**
+ * Get the total file size of a file slice in parquet format.
+ * For the log file, we need to convert its size to the estimated size in the parquet format in a certain proportion
+ */
+ public static long getTotalFileSizeAsParquetFormat(FileSlice fileSlice, double logFileFraction) {
+ long logFileSize = convertLogFilesSizeToExpectedParquetSize(fileSlice.getLogFiles().collect(Collectors.toList()), logFileFraction);
+ return fileSlice.getBaseFile().isPresent() ? fileSlice.getBaseFile().get().getFileSize() + logFileSize
+ : logFileSize;
+ }
+
+ private static long convertLogFilesSizeToExpectedParquetSize(List<HoodieLogFile> hoodieLogFiles, double logFileFraction) {
Review Comment:
Log format can be Avro, Parquet, or HFile. Could this be generalized instead of being fixated on Parquet?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -169,26 +169,29 @@ case class HoodieFileIndex(spark: SparkSession,
val prunedPartitionsAndFilteredFileSlices = filterFileSlices(dataFilters, partitionFilters).map {
case (partitionOpt, fileSlices) =>
if (shouldEmbedFileSlices) {
- val baseFileStatusesAndLogFileOnly: Seq[FileStatus] = fileSlices.map(slice => {
- if (slice.getBaseFile.isPresent) {
+ val logFileEstimationFraction = options.getOrElse(HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.key(),
+ HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.defaultValue()).toDouble
+ // 1. Generate a disguised representative file for each file slice, which spark uses to optimize rdd partition parallelism based on data such as file size
+ // For file slice only has base file, we directly use the base file size as representative file size
+ // For file slice has log file, we estimate the representative file size based on the log file size and option(base file) size
+ val representFiles = fileSlices.map(slice => {
+ val estimationFileSize = FileSliceUtils.getTotalFileSizeAsParquetFormat(slice, logFileEstimationFraction)
+ val fileInfo = if (slice.getBaseFile.isPresent) {
slice.getBaseFile.get().getPathInfo
- } else if (slice.hasLogFiles) {
- slice.getLogFiles.findAny().get().getPathInfo
} else {
- null
+ slice.getLogFiles.findAny().get().getPathInfo
}
- }).filter(slice => slice != null)
- .map(fileInfo => new FileStatus(fileInfo.getLength, fileInfo.isDirectory, 0, fileInfo.getBlockSize,
- fileInfo.getModificationTime, new Path(fileInfo.getPath.toUri)))
+ new FileStatus(estimationFileSize, fileInfo.isDirectory, 0, fileInfo.getBlockSize, fileInfo.getModificationTime, new Path(fileInfo.getPath.toUri))
Review Comment:
Does this affect reading the file because the file status is manipulated? Is there a different way of letting Spark know the file / partitioned file size estimation?
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileSliceUtils.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class FileSliceUtils {
+
+ /**
+ * Get the total file size of a file slice in parquet format.
+ * For the log file, we need to convert its size to the estimated size in the parquet format in a certain proportion
+ */
+ public static long getTotalFileSizeAsParquetFormat(FileSlice fileSlice, double logFileFraction) {
Review Comment:
Similar on the base file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2030655566
##########
hudi-common/src/main/java/org/apache/hudi/common/util/FileSliceUtils.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.model.FileSlice;
+import org.apache.hudi.common.model.HoodieLogFile;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+public class FileSliceUtils {
+
+ /**
+ * Get the total file size of a file slice in parquet format.
+ * For the log file, we need to convert its size to the estimated size in the parquet format in a certain proportion
+ */
+ public static long getTotalFileSizeAsParquetFormat(FileSlice fileSlice, double logFileFraction) {
+ long logFileSize = convertLogFilesSizeToExpectedParquetSize(fileSlice.getLogFiles().collect(Collectors.toList()), logFileFraction);
+ return fileSlice.getBaseFile().isPresent() ? fileSlice.getBaseFile().get().getFileSize() + logFileSize
+ : logFileSize;
+ }
+
+ private static long convertLogFilesSizeToExpectedParquetSize(List<HoodieLogFile> hoodieLogFiles, double logFileFraction) {
Review Comment:
> Log format can be Avro, Parquet, or HFile. Could this be generalized instead of being fixated on Parquet?
Now it is not fixed, `logFileFraction` comes from `HoodieStorageConfig#LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION`, users can change this value according to their own log format and the actual data pattern
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2774234034
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* e522af2c53e6db57bcd88e4249f86e01fb12a8e2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585)
* ad42cecbff2da69e2ce4bb0f17f9246d62709bb6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2024135479
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java:
##########
@@ -89,4 +91,15 @@ public class HoodieReaderConfig extends HoodieConfig {
"hoodie.write.record.merge.custom.implementation.classes";
public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
"hoodie.datasource.write.record.merger.impls";
+
+ public static final ConfigProperty<Double> LOG_FILE_TO_PARQUET_FORMAT_SIZE_ESTIMATION_FRACTION = ConfigProperty
+ .key("hoodie.logfile.to.parquet.format.size.estimation.fraction")
Review Comment:
Why not just use `config.getLogFileToParquetCompressionRatio()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2024154076
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -158,15 +158,20 @@ class HoodieFileGroupReaderBasedParquetFileFormat(tablePath: String,
file.partitionValues match {
// Snapshot or incremental queries.
case fileSliceMapping: HoodiePartitionFileSliceMapping =>
- val filegroupName = FSUtils.getFileIdFromFilePath(sparkAdapter
+ val fileGroupName = FSUtils.getFileIdFromFilePath(sparkAdapter
.getSparkPartitionedFileUtils.getPathFromPartitionedFile(file))
- fileSliceMapping.getSlice(filegroupName) match {
- case Some(fileSlice) if !isCount && (requiredSchema.nonEmpty || fileSlice.getLogFiles.findAny().isPresent) =>
Review Comment:
> why remove the `requiredSchema.nonEmpty` check.
<img width="681" alt="image" src="https://github.com/user-attachments/assets/9d39ebe7-4a22-4d26-b52e-3aee47a5eb97" />
This check has not been removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2038599931
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -169,26 +169,29 @@ case class HoodieFileIndex(spark: SparkSession,
val prunedPartitionsAndFilteredFileSlices = filterFileSlices(dataFilters, partitionFilters).map {
case (partitionOpt, fileSlices) =>
if (shouldEmbedFileSlices) {
- val baseFileStatusesAndLogFileOnly: Seq[FileStatus] = fileSlices.map(slice => {
- if (slice.getBaseFile.isPresent) {
+ val logFileEstimationFraction = options.getOrElse(HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.key(),
+ HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.defaultValue()).toDouble
+ // 1. Generate a disguised representative file for each file slice, which spark uses to optimize rdd partition parallelism based on data such as file size
+ // For file slice only has base file, we directly use the base file size as representative file size
+ // For file slice has log file, we estimate the representative file size based on the log file size and option(base file) size
+ val representFiles = fileSlices.map(slice => {
Review Comment:
delegateFiles
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771489346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 33ee1f487c6a97ace1c82caa6cf4019ec899b568 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579)
* 98f8b5a7964fc63138713c8dd5abdaec35452c50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2819000578
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818997991
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2819967414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2026330349
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -178,8 +183,8 @@ class HoodieFileGroupReaderBasedParquetFileFormat(tablePath: String,
internalSchemaOpt,
metaClient,
props,
- file.start,
- file.length,
+ 0,
Review Comment:
> then why the start is constant 0?
Please see the latest commit, `start` is still `file.start`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2024452438
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java:
##########
@@ -89,4 +91,15 @@ public class HoodieReaderConfig extends HoodieConfig {
"hoodie.write.record.merge.custom.implementation.classes";
public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
"hoodie.datasource.write.record.merger.impls";
+
+ public static final ConfigProperty<Double> LOG_FILE_TO_PARQUET_FORMAT_SIZE_ESTIMATION_FRACTION = ConfigProperty
+ .key("hoodie.logfile.to.parquet.format.size.estimation.fraction")
Review Comment:
> the write config is shared with the reader.
Done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818061730
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017)
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818294558
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-3103937748
> This is supposed to be
yes，a typo mistake :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2038600409
##########
hudi-common/src/main/java/org/apache/hudi/common/model/FileSlice.java:
##########
@@ -186,4 +186,26 @@ public boolean equals(Object o) {
public int hashCode() {
return Objects.hash(fileGroupId, baseInstantTime);
}
+
+ /**
+ * Get the total file size of a file slice similar on the base file.
+ * For the log file, we need to convert its size to the estimated size similar on the base file in a certain proportion
+ */
+ public long getTotalFileSizeSimilarOnBaseFile(double logFileFraction) {
+ long logFileSize = convertLogFilesSizeSimilarOnBaseFile(logFileFraction);
+ return getBaseFile().isPresent() ? getBaseFile().get().getFileSize() + logFileSize : logFileSize;
+ }
+
+ private long convertLogFilesSizeSimilarOnBaseFile(double logFileFraction) {
Review Comment:
still think the original method name is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2817979877
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 3c1fdea4346fadd861101a4c3d1709fd1909734c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006)
* 025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2052199976
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkPartitionFileUtils.scala:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi
+
+import org.apache.hudi.BaseHoodieTableFileIndex.PartitionPath
+import org.apache.hudi.common.config.HoodieStorageConfig
+import org.apache.hudi.common.model.FileSlice
+
+import org.apache.hadoop.fs.{FileStatus, Path}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources.PartitionDirectory
+
+object SparkPartitionFileUtils extends SparkAdapterSupport {
Review Comment:
> `PartitionDirectoryConverter`?
done~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2819143334
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2821156023
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
}, {
"hash" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059",
"triggerID" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"triggerType" : "PUSH"
}, {
"hash" : "1375df06346952de4af3d20daf22b3525d48723f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1375df06346952de4af3d20daf22b3525d48723f",
"triggerType" : "PUSH"
}, {
"hash" : "66a4d7039df6b3d709019fc1699e98f6a5c70c3f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5068",
"triggerID" : "66a4d7039df6b3d709019fc1699e98f6a5c70c3f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 3c414f6e97dd1184138acad1e0ea96a63e3f99b7 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059)
* 1375df06346952de4af3d20daf22b3525d48723f UNKNOWN
* 66a4d7039df6b3d709019fc1699e98f6a5c70c3f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5068)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2029620890
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/deltacommit/SparkUpsertDeltaCommitPartitioner.java:
##########
@@ -73,13 +74,13 @@ protected List<SmallFile> getSmallFiles(String partitionPath) {
if (smallFileSlice.getBaseFile().isPresent()) {
HoodieBaseFile baseFile = smallFileSlice.getBaseFile().get();
sf.location = new HoodieRecordLocation(baseFile.getCommitTime(), baseFile.getFileId());
- sf.sizeBytes = getTotalFileSize(smallFileSlice);
+ sf.sizeBytes = FileSliceUtils.getTotalFileSizeAsParquetFormat(smallFileSlice, config.getLogFileToParquetCompressionRatio());
Review Comment:
Can be moved to class `FileSlice`, and keep the comment of method: `convertLogFilesSizeToExpectedParquetSize`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2774205712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* e522af2c53e6db57bcd88e4249f86e01fb12a8e2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585)
* ad42cecbff2da69e2ce4bb0f17f9246d62709bb6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2025922724
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -178,8 +183,8 @@ class HoodieFileGroupReaderBasedParquetFileFormat(tablePath: String,
internalSchemaOpt,
metaClient,
props,
- file.start,
- file.length,
+ 0,
Review Comment:
> can you elaborate this change?
<img width="927" alt="image" src="https://github.com/user-attachments/assets/2315313f-cb52-42f0-b02e-4f0eb5e3c325" />
These two arguments are provided to the file group reader to tell it the start location and length of the base file. This value used to be taken directly from the `PartitiondFile` because when there is a base file in the file slice, The actual size of the base file is used as the length of the `PartitiondFile`. Now the `PartitiondFile` is a representative file of the file slice and is not the length of the actual base file, so we need to get the actual length out of the base file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2024173689
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -158,15 +158,20 @@ class HoodieFileGroupReaderBasedParquetFileFormat(tablePath: String,
file.partitionValues match {
// Snapshot or incremental queries.
case fileSliceMapping: HoodiePartitionFileSliceMapping =>
- val filegroupName = FSUtils.getFileIdFromFilePath(sparkAdapter
+ val fileGroupName = FSUtils.getFileIdFromFilePath(sparkAdapter
.getSparkPartitionedFileUtils.getPathFromPartitionedFile(file))
- fileSliceMapping.getSlice(filegroupName) match {
- case Some(fileSlice) if !isCount && (requiredSchema.nonEmpty || fileSlice.getLogFiles.findAny().isPresent) =>
Review Comment:
> why remove the `requiredSchema.nonEmpty` check.
Hi, I have rolled back these changes, I have kept the original logic as much as possible, and only changed the logic for creating representative files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771379314
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1db2dd895de991a9285a33bfff9c3d7cab7a26 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575)
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2024135785
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -158,15 +158,20 @@ class HoodieFileGroupReaderBasedParquetFileFormat(tablePath: String,
file.partitionValues match {
// Snapshot or incremental queries.
case fileSliceMapping: HoodiePartitionFileSliceMapping =>
- val filegroupName = FSUtils.getFileIdFromFilePath(sparkAdapter
+ val fileGroupName = FSUtils.getFileIdFromFilePath(sparkAdapter
.getSparkPartitionedFileUtils.getPathFromPartitionedFile(file))
- fileSliceMapping.getSlice(filegroupName) match {
- case Some(fileSlice) if !isCount && (requiredSchema.nonEmpty || fileSlice.getLogFiles.findAny().isPresent) =>
Review Comment:
why remove the `requiredSchema.nonEmpty` check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818298628
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2817577238
> @TheR1sing3un Thanks for the contribution, do we have some diagram to illustrage the files balance changes before/after the patch?
I added a test to verify the balance, please have a look at : `TestSparkPartitionFileUtils`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818862908
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2024128172
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -169,26 +169,24 @@ case class HoodieFileIndex(spark: SparkSession,
val prunedPartitionsAndFilteredFileSlices = filterFileSlices(dataFilters, partitionFilters).map {
case (partitionOpt, fileSlices) =>
if (shouldEmbedFileSlices) {
- val baseFileStatusesAndLogFileOnly: Seq[FileStatus] = fileSlices.map(slice => {
- if (slice.getBaseFile.isPresent) {
+ val logFileEstimationFraction = HoodieReaderConfig.getLogFileToParquetFormatSizeEstimationFraction(options.asJava)
+ // 1. Generate a disguised representative file for each file slice, which spark uses to optimize rdd partition parallelism based on data such as file size
+ // For file slice only has base file, we directly use the base file size as representative file size
+ // For file slice has log file, we estimate the representative file size based on the log file size and option(base file) size
+ val representFiles = fileSlices.map(slice => {
+ val estimationFileSize = FileSliceUtils.getTotalFileSizeAsParquetFormat(slice, logFileEstimationFraction)
+ val fileInfo = if (slice.getBaseFile.isPresent) {
slice.getBaseFile.get().getPathInfo
- } else if (slice.hasLogFiles) {
- slice.getLogFiles.findAny().get().getPathInfo
} else {
- null
+ slice.getLogFiles.findAny().get().getPathInfo
}
- }).filter(slice => slice != null)
- .map(fileInfo => new FileStatus(fileInfo.getLength, fileInfo.isDirectory, 0, fileInfo.getBlockSize,
- fileInfo.getModificationTime, new Path(fileInfo.getPath.toUri)))
- val c = fileSlices.filter(f => f.hasLogFiles || f.hasBootstrapBase).foldLeft(Map[String, FileSlice]()) { (m, f) => m + (f.getFileId -> f) }
- if (c.nonEmpty) {
Review Comment:
It does not look right to remove the if-else check, there are some special handlings in `HoodieSpark35PartitionedFileUtils.buildReaderWithPartitionValues` as regard to the special `HoodiePartitionFileSliceMapping` without log files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2024318657
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java:
##########
@@ -89,4 +91,15 @@ public class HoodieReaderConfig extends HoodieConfig {
"hoodie.write.record.merge.custom.implementation.classes";
public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
"hoodie.datasource.write.record.merger.impls";
+
+ public static final ConfigProperty<Double> LOG_FILE_TO_PARQUET_FORMAT_SIZE_ESTIMATION_FRACTION = ConfigProperty
+ .key("hoodie.logfile.to.parquet.format.size.estimation.fraction")
Review Comment:
> Is it unreasonable for us to use a write client-related parameter when reading?
the write config is shared with the reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2030659071
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala:
##########
@@ -169,26 +169,29 @@ case class HoodieFileIndex(spark: SparkSession,
val prunedPartitionsAndFilteredFileSlices = filterFileSlices(dataFilters, partitionFilters).map {
case (partitionOpt, fileSlices) =>
if (shouldEmbedFileSlices) {
- val baseFileStatusesAndLogFileOnly: Seq[FileStatus] = fileSlices.map(slice => {
- if (slice.getBaseFile.isPresent) {
+ val logFileEstimationFraction = options.getOrElse(HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.key(),
+ HoodieStorageConfig.LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION.defaultValue()).toDouble
+ // 1. Generate a disguised representative file for each file slice, which spark uses to optimize rdd partition parallelism based on data such as file size
+ // For file slice only has base file, we directly use the base file size as representative file size
+ // For file slice has log file, we estimate the representative file size based on the log file size and option(base file) size
+ val representFiles = fileSlices.map(slice => {
+ val estimationFileSize = FileSliceUtils.getTotalFileSizeAsParquetFormat(slice, logFileEstimationFraction)
+ val fileInfo = if (slice.getBaseFile.isPresent) {
slice.getBaseFile.get().getPathInfo
- } else if (slice.hasLogFiles) {
- slice.getLogFiles.findAny().get().getPathInfo
} else {
- null
+ slice.getLogFiles.findAny().get().getPathInfo
}
- }).filter(slice => slice != null)
- .map(fileInfo => new FileStatus(fileInfo.getLength, fileInfo.isDirectory, 0, fileInfo.getBlockSize,
- fileInfo.getModificationTime, new Path(fileInfo.getPath.toUri)))
+ new FileStatus(estimationFileSize, fileInfo.isDirectory, 0, fileInfo.getBlockSize, fileInfo.getModificationTime, new Path(fileInfo.getPath.toUri))
Review Comment:
> Does this affect reading the file because the file status is manipulated? Is there a different way of letting Spark know the file / partitioned file size estimation?
The actual read process is also controlled by hudi code. spark only does some parallelism related optimization through this file status, and the actual read is still controlled by hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771875587
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 98f8b5a7964fc63138713c8dd5abdaec35452c50 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2772132951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 98f8b5a7964fc63138713c8dd5abdaec35452c50 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582)
* e522af2c53e6db57bcd88e4249f86e01fb12a8e2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2772984879
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* e522af2c53e6db57bcd88e4249f86e01fb12a8e2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771454589
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1db2dd895de991a9285a33bfff9c3d7cab7a26 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575)
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 33ee1f487c6a97ace1c82caa6cf4019ec899b568 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771975587
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 98f8b5a7964fc63138713c8dd5abdaec35452c50 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582)
* e522af2c53e6db57bcd88e4249f86e01fb12a8e2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771370445
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1db2dd895de991a9285a33bfff9c3d7cab7a26 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818296189
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017)
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818512100
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2819162754
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2819160908
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2819324051
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2817577132
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 9372b4ba21c1286f378122a6f7975a620c5adca3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685)
* 3c1fdea4346fadd861101a4c3d1709fd1909734c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #13070:
URL: https://github.com/apache/hudi/pull/13070#discussion_r2024170453
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java:
##########
@@ -89,4 +91,15 @@ public class HoodieReaderConfig extends HoodieConfig {
"hoodie.write.record.merge.custom.implementation.classes";
public static final String RECORD_MERGE_IMPL_CLASSES_DEPRECATED_WRITE_CONFIG_KEY =
"hoodie.datasource.write.record.merger.impls";
+
+ public static final ConfigProperty<Double> LOG_FILE_TO_PARQUET_FORMAT_SIZE_ESTIMATION_FRACTION = ConfigProperty
+ .key("hoodie.logfile.to.parquet.format.size.estimation.fraction")
Review Comment:
> Why not just use `config.getLogFileToParquetCompressionRatio()`?
`HoodieStorageConfig#LOGFILE_TO_PARQUET_COMPRESSION_RATIO_FRACTION` belongs to the group `WRITE_CLIENT`.
<img width="543" alt="image" src="https://github.com/user-attachments/assets/1342b79a-2094-4f9d-9dc7-1fe94687a99a" />
Is it unreasonable for us to use a write client-related parameter when reading? By the way, this also raises another issue, many of the parameters that we use should be common to both read and write, but we put them in write config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2782526506
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* ad42cecbff2da69e2ce4bb0f17f9246d62709bb6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596)
* 9372b4ba21c1286f378122a6f7975a620c5adca3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2783782455
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 9372b4ba21c1286f378122a6f7975a620c5adca3 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2818058994
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 3c1fdea4346fadd861101a4c3d1709fd1909734c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006)
* 025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017)
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2820535071
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5041",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
}, {
"hash" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055",
"triggerID" : "0c9215910b27eb10c946bb72819b8ebe75129ca3",
"triggerType" : "PUSH"
}, {
"hash" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059",
"triggerID" : "3c414f6e97dd1184138acad1e0ea96a63e3f99b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* 0c9215910b27eb10c946bb72819b8ebe75129ca3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5055)
* 3c414f6e97dd1184138acad1e0ea96a63e3f99b7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5059)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2819983137
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
}, {
"hash" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4596",
"triggerID" : "ad42cecbff2da69e2ce4bb0f17f9246d62709bb6",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4680",
"triggerID" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"triggerType" : "PUSH"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4682",
"triggerID" : "2783062162",
"triggerType" : "MANUAL"
}, {
"hash" : "9372b4ba21c1286f378122a6f7975a620c5adca3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4685",
"triggerID" : "2783342263",
"triggerType" : "MANUAL"
}, {
"hash" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5006",
"triggerID" : "3c1fdea4346fadd861101a4c3d1709fd1909734c",
"triggerType" : "PUSH"
}, {
"hash" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5017",
"triggerID" : "025bd5bd5386ec951000a7c2c6dcdfd2b7ed3659",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"triggerType" : "PUSH"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021",
"triggerID" : "2818294558",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029",
"triggerID" : "2818509889",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031",
"triggerID" : "2818997991",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032",
"triggerID" : "2819160908",
"triggerType" : "MANUAL"
}, {
"hash" : "d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038",
"triggerID" : "2819883849",
"triggerType" : "MANUAL"
}, {
"hash" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "930b3491aece079a7f63a4fec22ed2a667a0d606",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* d7507d70b4a2aeafa25b5c8ecbfbc18c95e4768c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5025) Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5021) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5029) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5031) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5032) Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5038)
* 930b3491aece079a7f63a4fec22ed2a667a0d606 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #13070:
URL: https://github.com/apache/hudi/pull/13070
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2772697208
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
}, {
"hash" : "91837a853acbec30703aba8727dc83b4cff84148",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91837a853acbec30703aba8727dc83b4cff84148",
"triggerType" : "PUSH"
}, {
"hash" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4579",
"triggerID" : "33ee1f487c6a97ace1c82caa6cf4019ec899b568",
"triggerType" : "PUSH"
}, {
"hash" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4582",
"triggerID" : "98f8b5a7964fc63138713c8dd5abdaec35452c50",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583",
"triggerID" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"triggerType" : "PUSH"
}, {
"hash" : "e522af2c53e6db57bcd88e4249f86e01fb12a8e2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585",
"triggerID" : "2772692463",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 91837a853acbec30703aba8727dc83b4cff84148 UNKNOWN
* e522af2c53e6db57bcd88e4249f86e01fb12a8e2 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4583) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4585)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2772692463
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-3103929611
This is supposed to be https://issues.apache.org/jira/browse/HUDI-9250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9205] Introduce a representative file containing the estimated total size of file slice [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13070:
URL: https://github.com/apache/hudi/pull/13070#issuecomment-2771372151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575",
"triggerID" : "fd1db2dd895de991a9285a33bfff9c3d7cab7a26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fd1db2dd895de991a9285a33bfff9c3d7cab7a26 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4575)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org