You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2025/03/20 22:44:22 UTC
[PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
linliu-code opened a new pull request, #13008:
URL: https://github.com/apache/hudi/pull/13008
### Change Logs
Compared the stats from FG reader based compaction, and regular compaction stats, the `totalLogRecord` is not set properly. We fix this metric in this PR.
### Impact
The same stats as before.
### Risk level (write none, low medium or high below)
Low.
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2017485343
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -148,6 +157,34 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @ParameterizedTest
+ @EnumSource(RecordMergeMode.class)
+ public void testFileGroupReaderBasedCompaction(RecordMergeMode mergeMode) throws Exception {
Review Comment:
yes, after this change, it is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2742134944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa438bcdc09829a64cc9da3de9d635a8b56d758 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290)
* b00d2c37e80b3e2ebfde8400a39d62cda7028a6d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2744192590
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3825345315a79635a05f708f6e91e16ede9445f5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2742800882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b34e459d238dfaaad5c3babedca08bd2587a5334 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749104181
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3825345315a79635a05f708f6e91e16ede9445f5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317)
* b76c85eb3be46376606fba93b7ede9fccdf3550c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2758795750
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
}, {
"hash" : "e5060d759e2edca2008592bfceac3531a63338be",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e5060d759e2edca2008592bfceac3531a63338be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e988eaf68e614685207e68d97b2f30352602a50 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477)
* e5060d759e2edca2008592bfceac3531a63338be UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749619584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b76c85eb3be46376606fba93b7ede9fccdf3550c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366)
* 46496dfb3d6aa3c95be06b806e3a07d09c9ecd20 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367)
* 4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749812566
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
}, {
"hash" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368)
* dbe3cac0cd8f52209465e467955424f22b8bbc4c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749762796
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2758998174
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
}, {
"hash" : "e5060d759e2edca2008592bfceac3531a63338be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495",
"triggerID" : "e5060d759e2edca2008592bfceac3531a63338be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5060d759e2edca2008592bfceac3531a63338be Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2757259463
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e988eaf68e614685207e68d97b2f30352602a50 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2741828054
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa438bcdc09829a64cc9da3de9d635a8b56d758 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2741927655
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa438bcdc09829a64cc9da3de9d635a8b56d758 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2006774236
##########
hudi-client/hudi-java-client/src/test/java/org/apache/hudi/hadoop/TestHoodieFileGroupReaderOnHive.java:
##########
@@ -199,6 +201,12 @@ public void commitToTable(List<HoodieRecord> recordList, String operation, Map<S
}
}
+ @ParameterizedTest
Review Comment:
Override to skip since 1. Hive does not support fg reader based compaction; 2. The "commitToTable" function disables compaction. So no need to take the trouble to enable it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2008527592
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -250,6 +285,28 @@ private Map<String, String> getCommonConfigs(RecordMergeMode recordMergeMode) {
return configMapping;
}
+ private void validateCompactionStats(StorageConfiguration<?> storageConf,
+ String tablePath) {
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(storageConf, tablePath);
+ HoodieTimeline activeTimeline = metaClient.getActiveTimeline();
+ List<HoodieInstant> compactionInstants = activeTimeline
+ .getCommitsAndCompactionTimeline()
+ .getInstants()
+ .stream().filter(i -> i.getAction().equals(COMMIT_ACTION))
+ .collect(Collectors.toList());
+ assertFalse(compactionInstants.isEmpty());
+
+ try {
+ // Validate if total log records is updated.
+ // More validation can be added here if needed.
+ long totalLogRecords = activeTimeline
+ .readCommitMetadata(compactionInstants.get(0)).getTotalLogRecordsCompacted();
Review Comment:
Will add a few more metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2744018047
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b34e459d238dfaaad5c3babedca08bd2587a5334 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304)
* 3825345315a79635a05f708f6e91e16ede9445f5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749814480
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
}, {
"hash" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370",
"triggerID" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368)
* dbe3cac0cd8f52209465e467955424f22b8bbc4c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2759089134
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
}, {
"hash" : "e5060d759e2edca2008592bfceac3531a63338be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495",
"triggerID" : "e5060d759e2edca2008592bfceac3531a63338be",
"triggerType" : "PUSH"
}, {
"hash" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5060d759e2edca2008592bfceac3531a63338be Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495)
* 2c7cc85068f6d03e76ede062fe70c559f87e65b4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2763138499
Another one #13047 contains this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2742571272
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b00d2c37e80b3e2ebfde8400a39d62cda7028a6d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298)
* b34e459d238dfaaad5c3babedca08bd2587a5334 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2741829932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa438bcdc09829a64cc9da3de9d635a8b56d758 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2015505409
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -148,6 +157,34 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @ParameterizedTest
+ @EnumSource(RecordMergeMode.class)
+ public void testFileGroupReaderBasedCompaction(RecordMergeMode mergeMode) throws Exception {
Review Comment:
file group reader exposes readStats, but the total number of log records is always 0 since it does not know the log record number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2756993039
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475)
* 8e988eaf68e614685207e68d97b2f30352602a50 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749107916
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3825345315a79635a05f708f6e91e16ede9445f5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317)
* b76c85eb3be46376606fba93b7ede9fccdf3550c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749623906
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46496dfb3d6aa3c95be06b806e3a07d09c9ecd20 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367)
* 4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749314909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b76c85eb3be46376606fba93b7ede9fccdf3550c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2744021110
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b34e459d238dfaaad5c3babedca08bd2587a5334 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304)
* 3825345315a79635a05f708f6e91e16ede9445f5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749516331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b76c85eb3be46376606fba93b7ede9fccdf3550c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366)
* 46496dfb3d6aa3c95be06b806e3a07d09c9ecd20 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749926769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
}, {
"hash" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370",
"triggerID" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dbe3cac0cd8f52209465e467955424f22b8bbc4c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2742136532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5fa438bcdc09829a64cc9da3de9d635a8b56d758 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290)
* b00d2c37e80b3e2ebfde8400a39d62cda7028a6d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2008434375
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -250,6 +285,28 @@ private Map<String, String> getCommonConfigs(RecordMergeMode recordMergeMode) {
return configMapping;
}
+ private void validateCompactionStats(StorageConfiguration<?> storageConf,
+ String tablePath) {
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(storageConf, tablePath);
+ HoodieTimeline activeTimeline = metaClient.getActiveTimeline();
+ List<HoodieInstant> compactionInstants = activeTimeline
+ .getCommitsAndCompactionTimeline()
+ .getInstants()
+ .stream().filter(i -> i.getAction().equals(COMMIT_ACTION))
+ .collect(Collectors.toList());
+ assertFalse(compactionInstants.isEmpty());
+
+ try {
+ // Validate if total log records is updated.
+ // More validation can be added here if needed.
+ long totalLogRecords = activeTimeline
+ .readCommitMetadata(compactionInstants.get(0)).getTotalLogRecordsCompacted();
Review Comment:
I understand thats the only gap, but when we are writing a new test to validate the stats, why just validate just log records. lets try to validate all stats w/ some expected value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2759313586
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
}, {
"hash" : "e5060d759e2edca2008592bfceac3531a63338be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495",
"triggerID" : "e5060d759e2edca2008592bfceac3531a63338be",
"triggerType" : "PUSH"
}, {
"hash" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4501",
"triggerID" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"triggerType" : "PUSH"
}, {
"hash" : "423006aff1c2d95f2732a2a9bfad785c105ec8dc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4502",
"triggerID" : "423006aff1c2d95f2732a2a9bfad785c105ec8dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c7cc85068f6d03e76ede062fe70c559f87e65b4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4501)
* 423006aff1c2d95f2732a2a9bfad785c105ec8dc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4502)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2755142242
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f31bc8eb92a06bafdd0771ae5efd342bcf637aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2759110536
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
}, {
"hash" : "e5060d759e2edca2008592bfceac3531a63338be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495",
"triggerID" : "e5060d759e2edca2008592bfceac3531a63338be",
"triggerType" : "PUSH"
}, {
"hash" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4501",
"triggerID" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"triggerType" : "PUSH"
}, {
"hash" : "423006aff1c2d95f2732a2a9bfad785c105ec8dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "423006aff1c2d95f2732a2a9bfad785c105ec8dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5060d759e2edca2008592bfceac3531a63338be Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495)
* 2c7cc85068f6d03e76ede062fe70c559f87e65b4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4501)
* 423006aff1c2d95f2732a2a9bfad785c105ec8dc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2759113899
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
}, {
"hash" : "e5060d759e2edca2008592bfceac3531a63338be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495",
"triggerID" : "e5060d759e2edca2008592bfceac3531a63338be",
"triggerType" : "PUSH"
}, {
"hash" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4501",
"triggerID" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"triggerType" : "PUSH"
}, {
"hash" : "423006aff1c2d95f2732a2a9bfad785c105ec8dc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "423006aff1c2d95f2732a2a9bfad785c105ec8dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c7cc85068f6d03e76ede062fe70c559f87e65b4 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4501)
* 423006aff1c2d95f2732a2a9bfad785c105ec8dc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2006774236
##########
hudi-client/hudi-java-client/src/test/java/org/apache/hudi/hadoop/TestHoodieFileGroupReaderOnHive.java:
##########
@@ -199,6 +201,12 @@ public void commitToTable(List<HoodieRecord> recordList, String operation, Map<S
}
}
+ @ParameterizedTest
Review Comment:
Hive does not support fg reader compaction. Override to skip for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2742568799
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b00d2c37e80b3e2ebfde8400a39d62cda7028a6d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298)
* b34e459d238dfaaad5c3babedca08bd2587a5334 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2010860861
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -250,6 +287,38 @@ private Map<String, String> getCommonConfigs(RecordMergeMode recordMergeMode) {
return configMapping;
}
+ private void validateCompactionStats(StorageConfiguration<?> storageConf,
+ String tablePath) {
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(storageConf, tablePath);
+ HoodieTimeline activeTimeline = metaClient.getActiveTimeline();
+ List<HoodieInstant> compactionInstants = activeTimeline
+ .getCommitsAndCompactionTimeline()
+ .getInstants()
+ .stream().filter(i -> i.getAction().equals(COMMIT_ACTION))
+ .collect(Collectors.toList());
+ assertFalse(compactionInstants.isEmpty());
+
+ try {
+ // Validate if total log records is updated.
+ // More validation can be added here if needed.
+ HoodieCommitMetadata metadata = activeTimeline
+ .readCommitMetadata(compactionInstants.get(0));
+ assertTrue(metadata.getTotalLogRecordsCompacted() > 0);
Review Comment:
atleast for total log records, we should be able to assert exact value right?
for scan time, and other latencies, I understand, we cannot do exact match w/ our expected value.
but for total log records compacted, numInserts, numUpdates etc, we should assert for exact value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2010991046
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -250,6 +287,38 @@ private Map<String, String> getCommonConfigs(RecordMergeMode recordMergeMode) {
return configMapping;
}
+ private void validateCompactionStats(StorageConfiguration<?> storageConf,
+ String tablePath) {
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(storageConf, tablePath);
+ HoodieTimeline activeTimeline = metaClient.getActiveTimeline();
+ List<HoodieInstant> compactionInstants = activeTimeline
+ .getCommitsAndCompactionTimeline()
+ .getInstants()
+ .stream().filter(i -> i.getAction().equals(COMMIT_ACTION))
+ .collect(Collectors.toList());
+ assertFalse(compactionInstants.isEmpty());
+
+ try {
+ // Validate if total log records is updated.
+ // More validation can be added here if needed.
+ HoodieCommitMetadata metadata = activeTimeline
+ .readCommitMetadata(compactionInstants.get(0));
+ assertTrue(metadata.getTotalLogRecordsCompacted() > 0);
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2752709317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
}, {
"hash" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370",
"triggerID" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"triggerType" : "PUSH"
}, {
"hash" : "c6195191246eddf09e5e8a0034c4056522e7dcdc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4403",
"triggerID" : "c6195191246eddf09e5e8a0034c4056522e7dcdc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dbe3cac0cd8f52209465e467955424f22b8bbc4c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370)
* c6195191246eddf09e5e8a0034c4056522e7dcdc Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4403)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2752707550
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
}, {
"hash" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370",
"triggerID" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"triggerType" : "PUSH"
}, {
"hash" : "c6195191246eddf09e5e8a0034c4056522e7dcdc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6195191246eddf09e5e8a0034c4056522e7dcdc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* dbe3cac0cd8f52209465e467955424f22b8bbc4c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370)
* c6195191246eddf09e5e8a0034c4056522e7dcdc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2749514067
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b76c85eb3be46376606fba93b7ede9fccdf3550c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366)
* 46496dfb3d6aa3c95be06b806e3a07d09c9ecd20 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2752876583
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
}, {
"hash" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370",
"triggerID" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"triggerType" : "PUSH"
}, {
"hash" : "c6195191246eddf09e5e8a0034c4056522e7dcdc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4403",
"triggerID" : "c6195191246eddf09e5e8a0034c4056522e7dcdc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6195191246eddf09e5e8a0034c4056522e7dcdc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4403)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2742274834
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b00d2c37e80b3e2ebfde8400a39d62cda7028a6d Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2741905888
> isn't there any tests where we validate these stats. if not, can you write one.
sg, I verified manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2015500768
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -238,6 +243,7 @@ protected Option<Pair<Option<T>, Map<String, Object>>> doProcessNextDataRecord(T
Map<String, Object> metadata,
Pair<Option<T>, Map<String, Object>> existingRecordMetadataPair)
throws IOException {
+ totalLogRecords++;
Review Comment:
logRecordReader does not know the # of log records; it only knows the final records with distinct record keys. This number can only be collected inside the buffer when we read the blocks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2756873488
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f31bc8eb92a06bafdd0771ae5efd342bcf637aa Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438)
* 045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2015170723
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -148,6 +157,34 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @ParameterizedTest
+ @EnumSource(RecordMergeMode.class)
+ public void testFileGroupReaderBasedCompaction(RecordMergeMode mergeMode) throws Exception {
Review Comment:
The file group reader exposes `readStats`. Could you validate that as part of existing FG reader tests?
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -250,6 +287,42 @@ private Map<String, String> getCommonConfigs(RecordMergeMode recordMergeMode) {
return configMapping;
}
+ private void validateCompactionStats(StorageConfiguration<?> storageConf,
Review Comment:
We should not validate the compaction stats at the test in this layer. It should be validated in compaction functional tests.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -238,6 +243,7 @@ protected Option<Pair<Option<T>, Map<String, Object>>> doProcessNextDataRecord(T
Map<String, Object> metadata,
Pair<Option<T>, Map<String, Object>> existingRecordMetadataPair)
throws IOException {
+ totalLogRecords++;
Review Comment:
Why do we need this? The file group reader already does `readStats.setTotalLogRecords(logRecordReader.getTotalLogRecords())`.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -238,6 +243,7 @@ protected Option<Pair<Option<T>, Map<String, Object>>> doProcessNextDataRecord(T
Map<String, Object> metadata,
Pair<Option<T>, Map<String, Object>> existingRecordMetadataPair)
throws IOException {
+ totalLogRecords++;
Review Comment:
Should the counter be incremented inside the log reader instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2758801589
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
}, {
"hash" : "e5060d759e2edca2008592bfceac3531a63338be",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495",
"triggerID" : "e5060d759e2edca2008592bfceac3531a63338be",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8e988eaf68e614685207e68d97b2f30352602a50 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477)
* e5060d759e2edca2008592bfceac3531a63338be Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2759478847
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
}, {
"hash" : "e5060d759e2edca2008592bfceac3531a63338be",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495",
"triggerID" : "e5060d759e2edca2008592bfceac3531a63338be",
"triggerType" : "PUSH"
}, {
"hash" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4501",
"triggerID" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"triggerType" : "PUSH"
}, {
"hash" : "423006aff1c2d95f2732a2a9bfad785c105ec8dc",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4502",
"triggerID" : "423006aff1c2d95f2732a2a9bfad785c105ec8dc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 423006aff1c2d95f2732a2a9bfad785c105ec8dc Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4502)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2015512474
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -250,6 +287,42 @@ private Map<String, String> getCommonConfigs(RecordMergeMode recordMergeMode) {
return configMapping;
}
+ private void validateCompactionStats(StorageConfiguration<?> storageConf,
Review Comment:
I will move it to other compaction test class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2756880000
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475)
* 8e988eaf68e614685207e68d97b2f30352602a50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2016926543
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -148,6 +157,34 @@ public void testReadFileGroupInMergeOnReadTable(RecordMergeMode recordMergeMode,
}
}
+ @ParameterizedTest
+ @EnumSource(RecordMergeMode.class)
+ public void testFileGroupReaderBasedCompaction(RecordMergeMode mergeMode) throws Exception {
Review Comment:
Do you mean the issue is not fixed? Reading a file group with logs should now return non-zero number of log records, correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2756877413
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f31bc8eb92a06bafdd0771ae5efd342bcf637aa Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438)
* 045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475)
* 8e988eaf68e614685207e68d97b2f30352602a50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2759094599
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
}, {
"hash" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4475",
"triggerID" : "045c5bc7fae64a8ecfb608e32fdfd5aaefa3a2ec",
"triggerType" : "PUSH"
}, {
"hash" : "8e988eaf68e614685207e68d97b2f30352602a50",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4477",
"triggerID" : "8e988eaf68e614685207e68d97b2f30352602a50",
"triggerType" : "PUSH"
}, {
"hash" : "e5060d759e2edca2008592bfceac3531a63338be",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495",
"triggerID" : "e5060d759e2edca2008592bfceac3531a63338be",
"triggerType" : "PUSH"
}, {
"hash" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4501",
"triggerID" : "2c7cc85068f6d03e76ede062fe70c559f87e65b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e5060d759e2edca2008592bfceac3531a63338be Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4495)
* 2c7cc85068f6d03e76ede062fe70c559f87e65b4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4501)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13008:
URL: https://github.com/apache/hudi/pull/13008#discussion_r2010903701
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -250,6 +287,38 @@ private Map<String, String> getCommonConfigs(RecordMergeMode recordMergeMode) {
return configMapping;
}
+ private void validateCompactionStats(StorageConfiguration<?> storageConf,
+ String tablePath) {
+ HoodieTableMetaClient metaClient = HoodieTestUtils.createMetaClient(storageConf, tablePath);
+ HoodieTimeline activeTimeline = metaClient.getActiveTimeline();
+ List<HoodieInstant> compactionInstants = activeTimeline
+ .getCommitsAndCompactionTimeline()
+ .getInstants()
+ .stream().filter(i -> i.getAction().equals(COMMIT_ACTION))
+ .collect(Collectors.toList());
+ assertFalse(compactionInstants.isEmpty());
+
+ try {
+ // Validate if total log records is updated.
+ // More validation can be added here if needed.
+ HoodieCommitMetadata metadata = activeTimeline
+ .readCommitMetadata(compactionInstants.get(0));
+ assertTrue(metadata.getTotalLogRecordsCompacted() > 0);
Review Comment:
I was not sure if that is true. let me verify that and update if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2755131711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4290",
"triggerID" : "5fa438bcdc09829a64cc9da3de9d635a8b56d758",
"triggerType" : "PUSH"
}, {
"hash" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4298",
"triggerID" : "b00d2c37e80b3e2ebfde8400a39d62cda7028a6d",
"triggerType" : "PUSH"
}, {
"hash" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4304",
"triggerID" : "b34e459d238dfaaad5c3babedca08bd2587a5334",
"triggerType" : "PUSH"
}, {
"hash" : "3825345315a79635a05f708f6e91e16ede9445f5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4317",
"triggerID" : "3825345315a79635a05f708f6e91e16ede9445f5",
"triggerType" : "PUSH"
}, {
"hash" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4366",
"triggerID" : "b76c85eb3be46376606fba93b7ede9fccdf3550c",
"triggerType" : "PUSH"
}, {
"hash" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4367",
"triggerID" : "46496dfb3d6aa3c95be06b806e3a07d09c9ecd20",
"triggerType" : "PUSH"
}, {
"hash" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4368",
"triggerID" : "4880ec2bfc5d3c59a24c6ac679c5dbe7b01e4087",
"triggerType" : "PUSH"
}, {
"hash" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4370",
"triggerID" : "dbe3cac0cd8f52209465e467955424f22b8bbc4c",
"triggerType" : "PUSH"
}, {
"hash" : "c6195191246eddf09e5e8a0034c4056522e7dcdc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4403",
"triggerID" : "c6195191246eddf09e5e8a0034c4056522e7dcdc",
"triggerType" : "PUSH"
}, {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6195191246eddf09e5e8a0034c4056522e7dcdc Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4403)
* 9f31bc8eb92a06bafdd0771ae5efd342bcf637aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13008:
URL: https://github.com/apache/hudi/pull/13008#issuecomment-2755147950
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438",
"triggerID" : "9f31bc8eb92a06bafdd0771ae5efd342bcf637aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f31bc8eb92a06bafdd0771ae5efd342bcf637aa Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4438)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8635] Support totalLogRecord metric for FG reader based compaction [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code closed pull request #13008: [HUDI-8635] Support totalLogRecord metric for FG reader based compaction
URL: https://github.com/apache/hudi/pull/13008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org