You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "shangxinli (via GitHub)" <gi...@apache.org> on 2026/09/08 14:04:21 UTC
[PR] feat(metadata): report metadata table compaction health metrics [hudi]
shangxinli opened a new pull request, #19868:
URL: https://github.com/apache/hudi/pull/19868
### Describe the issue this Pull Request addresses
The metadata table reports `baseFileCount` and `logFileCount` per partition, but nothing reports whether MDT compaction is keeping up with the delta commits being written to it.
When compaction stops being scheduled, or fails quietly because `hoodie.metadata.write.fail.on.table.service.failures` is false and the exception is swallowed, log files accumulate against a fixed set of base files. Every metadata lookup then merges more log blocks per file slice. The first thing an operator actually observes is slower lookups and higher memory use on the read path, long after the point where compaction stopped. The per-partition counts do not surface it, because a table with a healthy file count and a table that has not compacted in a thousand delta commits can report identical `baseFileCount`.
### Summary and Changelog
Adds four gauges to `HoodieMetadataMetrics` so that a stalled MDT compaction is visible while it is still cheap to fix.
- `deltaCommitsSinceLastCompaction` — **completed** delta commits on the metadata table since the last completed compaction. Emitted from `HoodieBackedTableMetadataWriter.performTableServices()`. This counts the same instants that `ScheduleCompactionActionExecutor.getLatestDeltaCommitInfoSinceCompaction()` compares against `hoodie.compact.inline.max.delta.commits`. It is sampled *before* the compaction in the same method runs, so on a healthy table the gauge sawtooths: it peaks at that config value on the cycle where compaction fires, then drops. Alert on a multiple of the config value, not the value itself. A stalled compaction climbs past the peak without bound, which is the signal. Reset to 0 on the path where the metadata timeline has no completed delta commit yet, so a stale value is not left latched.
- `totalBaseFileCount` / `totalLogFileCount` — the existing per-partition counts summed over the enabled metadata partitions.
- `logToBaseFileRatioPercent` — log files per base file, scaled by 100 because `HoodieGauge` is long valued. 500 means five log files per base file.
The ratio returns 0 when there are no base files to divide by. That case is not healthy — a metadata table holding log files and no base file has never been compacted — which is why the two totals are emitted alongside it rather than the ratio alone.
Changelog:
- `HoodieMetadataMetrics`: four new metric-name constants; `updateDeltaCommitsSinceLastCompaction(long)`; `updateSizeMetrics()` now also emits the two totals and the ratio; `sumStat()` and `logToBaseFileRatioPercent()` extracted as package-private statics so the arithmetic is unit-testable without standing up a file system view.
- `HoodieBackedTableMetadataWriter.performTableServices()`: emits `deltaCommitsSinceLastCompaction`, reusing `CompactionUtils.getCompletedDeltaCommitsSinceLatestCompaction()`.
- New `TestHoodieMetadataMetrics` in `hudi-common`, covering gauge emission through a real `Metrics` instance with the `INMEMORY` reporter, the ×100 scaling, the zero-base-file case, and stat summation over enabled partitions only.
- Three cases added to `TestHoodieBackedTableMetadataWriter` covering the compacted, never-compacted, and empty-timeline paths. These run against a real `ActiveTimelineV2` rather than a static mock of `CompactionUtils`, so the completed-versus-inflight distinction is actually asserted: the main case has three completed delta commits and one inflight after the last compaction, and expects 3.
Three notes on choices reviewers may want to push back on, stated up front:
- The new gauges are **not** table-name prefixed, unlike `TABLE_SERVICE_EXECUTION_DURATION` in the same method. They follow the size-metric family instead. `Metrics.getBasePath()` strips the metadata-table suffix, so a metadata writer deliberately shares one registry with its own data-table writer, and distinct tables have distinct base paths — meaning cross-table collision is not possible and the prefix is not load-bearing here. The names are also distinct from the data-writer metric names in that shared registry. Happy to prefix them if the project prefers consistency within the method over consistency within the metric family.
- The delta-commit gauge is sampled before table services rather than re-sampled after a synchronous compaction. Re-sampling would give a cleaner post-compaction value, but needs another `reloadActiveTimeline()` on the write path, and `compactIfNecessary` is `void` and has three outcomes (skipped, delegated to the table service manager, compacted inline) so "did it actually compact" is not currently observable at the call site. Flagging it as a deliberate trade-off; if reviewers would rather pay the reload, it is a small follow-up.
- `sumStat` and `logToBaseFileRatioPercent` are package-private with `@VisibleForTesting` rather than private, so the arithmetic can be tested without constructing a `HoodieTableFileSystemView`. Can be inlined back to private if preferred.
No code was copied from another project.
### Impact
No public API change, no config change, no behavior change to writes or reads.
Cost per commit is negligible and adds no listing or scan. `deltaCommitsSinceLastCompaction` reuses `CompactionUtils.getCompletedDeltaCommitsSinceLatestCompaction()` over the already-loaded active timeline. The totals are summed from the stats map `updateSizeMetrics()` already builds, which is O(enabled partitions) — roughly seven entries.
Gauge cardinality is fixed: four new series per table, independent of partition or file count.
Users on a metrics reporter gain four new series. Users with metrics disabled see nothing, since every emission goes through the existing `Option<HoodieMetadataMetrics>` guard.
### Risk Level
low
Metrics-only change on paths that already emit gauges. The one behavioral edge is the early return in `performTableServices()` when the metadata timeline has no completed delta commit; the gauge is set to 0 before returning so the value is not left stale, and that path is covered by a test.
### Documentation Update
No new configs, so no config description update is required.
The four metric names could reasonably be added to the metadata-table section of the metrics page on the Hudi website. Happy to open the companion website PR if reviewers want them listed.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3991771395
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -651,4 +651,83 @@ void testPerformTableServicesWithFailureHandling(
// Verify metrics are incremented when there's a failure
verify(metrics, times(1)).incrementMetric(HoodieMetadataMetrics.PENDING_COMPACTIONS_FAILURES, 1);
}
+
+ @Test
+ void performTableServicesReportsCompletedDeltaCommitsSinceLastCompaction() throws Exception {
+ // A completed compaction, then three completed delta commits after it, plus one still inflight.
+ // The gauge must count the three completed instants only, so that it stays comparable to
+ // hoodie.compact.inline.max.delta.commits, which the compaction trigger evaluates the same way.
+ List<HoodieInstant> instants = new ArrayList<>();
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "001", "0011"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, "002", "0021"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "003", "0031"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "004", "0041"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "005", "0051"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, "006"));
+
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(instants)).performTableServices(Option.empty(), true);
+
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(3L);
+ }
+
+ @Test
+ void performTableServicesReportsAllDeltaCommitsWhenNeverCompacted() throws Exception {
Review Comment:
Folded into one `@ParameterizedTest` over `(instants, expected)`. Added a failure-path assertion as well, since the sample now runs in `finally`.
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -523,7 +523,7 @@ private static void setField(Object target, String name, Object value) throws Ex
}
@SuppressWarnings("deprecation")
- private HoodieActiveTimeline createMockTimeline(List<HoodieInstant> instants) {
+ private static HoodieActiveTimeline createMockTimeline(List<HoodieInstant> instants) {
Review Comment:
Reverted, unnecessary churn.
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -651,4 +651,83 @@ void testPerformTableServicesWithFailureHandling(
// Verify metrics are incremented when there's a failure
verify(metrics, times(1)).incrementMetric(HoodieMetadataMetrics.PENDING_COMPACTIONS_FAILURES, 1);
}
+
+ @Test
+ void performTableServicesReportsCompletedDeltaCommitsSinceLastCompaction() throws Exception {
+ // A completed compaction, then three completed delta commits after it, plus one still inflight.
+ // The gauge must count the three completed instants only, so that it stays comparable to
+ // hoodie.compact.inline.max.delta.commits, which the compaction trigger evaluates the same way.
Review Comment:
Right, `hoodie.metadata.compact.max.delta.commits` is the key operators actually set. Fixed.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
Review Comment:
Both of these are resolved by dropping the three size gauges rather than documenting the gating: only `deltaCommitsSinceLastCompaction` remains, and it is not gated on `hoodie.metadata.enable.detailed.metrics`. @voonhous thanks for pinning down the default and the derivability, that is what decided it.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +85,22 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges
+ // show the current shape of the metadata table, but not whether compaction is keeping up with it.
+ // Completed delta commits on the metadata table since the last completed compaction, sampled before
+ // table services run. On a healthy table this sawtooths up to hoodie.metadata.compact.max.delta.commits
+ // and falls back after each compaction, so alert on a multiple of that config rather than on the
+ // config value itself. A value that climbs past the peak and keeps going means metadata table
+ // compaction is not being scheduled or is failing.
+ public static final String STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION = "deltaCommitsSinceLastCompaction";
+ // Base and log file counts summed across all enabled metadata partitions.
+ public static final String STAT_TOTAL_BASE_FILE_COUNT = "totalBaseFileCount";
+ public static final String STAT_TOTAL_LOG_FILE_COUNT = "totalLogFileCount";
+ // Log files per base file across the whole metadata table, as a percentage, since gauges are long
+ // valued: 100 means one log file per base file, 500 means five. A rising value means readers must
Review Comment:
Moot now, those three constants are gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5667368691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
}, {
"hash" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048",
"triggerID" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"triggerType" : "PUSH"
}, {
"hash" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17095",
"triggerID" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* 6e2fcf0145352e8035fb38616a45473e243e58d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17095)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5742392506
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
}, {
"hash" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048",
"triggerID" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"triggerType" : "PUSH"
}, {
"hash" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17095",
"triggerID" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"triggerType" : "PUSH"
}, {
"hash" : "2dac8d64704793280240ec0aa2b4f0b122d713f8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2dac8d64704793280240ec0aa2b4f0b122d713f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* 6e2fcf0145352e8035fb38616a45473e243e58d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17095)
* 2dac8d64704793280240ec0aa2b4f0b122d713f8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5589881696
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19868?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `93.75000%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 78.02%. Comparing base ([`cebbd6a`](https://app.codecov.io/gh/apache/hudi/commit/cebbd6a0d56423383ffe5c8bd947db14c646e3d5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a522fe0`](https://app.codecov.io/gh/apache/hudi/commit/a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 7 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19868?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/19868?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | 80.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19868?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19868 +/- ##
============================================
- Coverage 78.33% 78.02% -0.32%
- Complexity 33943 34053 +110
============================================
Files 2543 2543
Lines 142035 143065 +1030
Branches 17246 17516 +270
============================================
+ Hits 111270 111625 +355
- Misses 23054 23654 +600
- Partials 7711 7786 +75
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.37% <100.00%> (-1.39%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.19% <80.00%> (-0.02%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.52% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.24% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.50% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.13% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.79% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.95% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.42% <ø> (-0.61%)` | :arrow_down: |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.81% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19868/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19868/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19868/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.09% <93.75%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19868?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...rg/apache/hudi/metadata/HoodieMetadataMetrics.java](https://app.codecov.io/gh/apache/hudi/pull/19868?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieMetadataMetrics.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0YWRhdGEvSG9vZGllTWV0YWRhdGFNZXRyaWNzLmphdmE=) | `92.42% <100.00%> (+1.51%)` | :arrow_up: |
| [...hudi/metadata/HoodieBackedTableMetadataWriter.java](https://app.codecov.io/gh/apache/hudi/pull/19868?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetadata%2FHoodieBackedTableMetadataWriter.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldGFkYXRhL0hvb2RpZUJhY2tlZFRhYmxlTWV0YWRhdGFXcml0ZXIuamF2YQ==) | `90.80% <80.00%> (-0.07%)` | :arrow_down: |
... and [8 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19868/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5640343074
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* bde2784f650ada62b14f34af553a53a28ef946e6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5742492435
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
}, {
"hash" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048",
"triggerID" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"triggerType" : "PUSH"
}, {
"hash" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17095",
"triggerID" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"triggerType" : "PUSH"
}, {
"hash" : "2dac8d64704793280240ec0aa2b4f0b122d713f8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17209",
"triggerID" : "2dac8d64704793280240ec0aa2b4f0b122d713f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* 6e2fcf0145352e8035fb38616a45473e243e58d3 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17095)
* 2dac8d64704793280240ec0aa2b4f0b122d713f8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17209)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3993618080
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +84,26 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges show
+ // the current shape of the metadata table, but not whether compaction is keeping up with it. This is
+ // the count of completed delta commits since the last completed compaction, counting the same instants
+ // that ScheduleCompactionActionExecutor compares against hoodie.metadata.compact.max.delta.commits.
+ //
+ // It is sampled after table services have run, so a healthy table reports close to zero once a
+ // compaction lands, and a table whose compaction is failing or is never scheduled keeps reporting a
+ // backlog at or above the threshold. Alert on it staying at or above the threshold across consecutive
+ // cycles rather than on a fixed multiple of it: the value does not grow without bound. Once the backlog
+ // reaches the threshold the archiver stops pinning the last compaction
+ // (CompactionUtils#getEarliestInstantToRetainForCompaction), MDT archival eventually removes it, and the
+ // count then plateaus near hoodie.keep.min.commits, which depends on the archival config as well.
+ //
+ // It is emitted from HoodieBackedTableMetadataWriter#performTableServices, so it covers table services
+ // driven through that path. Under hoodie.metadata.streaming.write.enabled, Flink runs MDT compaction in
+ // a separate pipeline that does not call it, and this gauge is not emitted there. Read it alongside
+ // TABLE_SERVICE_EXECUTION_STATUS and COMPACTION_FAILURES, which distinguish a failing compaction from
+ // table services having stopped running at all.
+ public static final String STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION = "deltaCommitsSinceLastCompaction";
+
private final transient MetricRegistry metricsRegistry;
Review Comment:
🤖 nit: the other metric name constants in this file (`LOG_COMPACTION_FAILURES`, `PENDING_COMPACTIONS_FAILURES`) are neither prefixed `STAT_` nor camelCase for the value string (`logcompaction_failures`). This new one breaks both conventions (`STAT_` prefix, `deltaCommitsSinceLastCompaction` camelCase) - might be worth aligning with the existing snake_case style and dropping the prefix for consistency.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5646648815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
}, {
"hash" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048",
"triggerID" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* bde2784f650ada62b14f34af553a53a28ef946e6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024)
* 9625882157b3236831467434f43bc9fe3be4cf1a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5743146096
Reopening to retrigger CI — test-flink-2 failed on ITTestDataStreamV2Write.testMergeOnReadWithCompaction, which is unrelated to this change (no hudi-flink files touched) and looks flaky.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5638190912
Thanks @voonhous, this was a genuinely useful review — I verified all four majors against the code and they all hold. Summary of the revision:
**Fixed**
- The gauge no longer freezes on the failure path. It is now sampled once in the `finally` block of `performTableServices()`. One refinement on your suggestion: `getActiveTimeline()` there is not enough either, since every caller passes `requiresTimelineRefresh=false` (`HoodieTableMetadataWriter.java:178`) and neither `runAnyPendingCompactions` nor `writeClient.compact` refreshes this meta client — so it samples off `reloadActiveTimeline()`, wrapped so it can neither mask a table service failure nor become one.
- Prefixed with the MDT table name so `CloudWatchReporter` no longer drops it.
- Reworded the plateau behavior. I did not adopt the "alert at 2x" guidance, because the plateau depends on the archival config as well as the compaction threshold — now that the sample is post-service, a healthy table returns to about zero each cycle, so the condition is "stays at or above the threshold across consecutive cycles".
**Scope cut**
Dropped `totalBaseFileCount`, `totalLogFileCount` and `logToBaseFileRatioPercent` rather than fixing each. Between your ratio-weighting case, the detailed-metrics default, and the fact that all three are arithmetic over per-partition gauges the same call already emits, they were not earning their place in a compaction-health PR. That resolves five of the comments at once, including the missing `updateSizeMetrics` coverage — though that gap is real and pre-existing, and I am happy to file it separately.
**Tests**
Folded into one `@ParameterizedTest`, unified onto a single setup helper, reverted the `static`, fixed the config key. Added three cases your review implied: completion-time ordering (`0015`/`0035`), compaction-last, and the failure path. I also added a case that pins the reload — the cached and reloaded timelines are stubbed to different values, so it fails if the sample goes back to `getActiveTimeline()`. Each of the three fixes was mutation-checked: reverting the reload fails 1 test, dropping the name prefix fails 10, and moving the sample out of `finally` fails 5.
**Not doing here**
- Emitting from the Flink streaming-write compaction pipeline. Real gap, confirmed at both guards — documented on the constant and left as a follow-up rather than widened into this PR.
- Wiring up or deleting `STAT_LAST_COMPACTION_TIMESTAMP`. Dead since 2020 and independent of this change. An age-since-compaction gauge is a good idea and I would rather propose it on its own.
30/30 tests pass locally in `hudi-client-common`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5638243188
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914)
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3961457864
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1543,8 +1543,20 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
.filterCompletedInstants()
.lastInstant();
if (!lastInstant.isPresent()) {
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(0L));
return;
}
+ // Report how far the metadata table has drifted from its last compaction, so that a stalled
+ // compaction is visible before the log to base file ratio degrades read performance. This is the
+ // backlog as it stands before the compaction below runs, so on a healthy table it peaks at
+ // hoodie.metadata.compact.max.delta.commits on the cycle that compacts and drops on the next one.
+ // It counts the same completed instants that ScheduleCompactionActionExecutor compares against
+ // that threshold, and is not re-sampled afterwards because doing so would need another timeline
+ // reload on the write path.
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(
Review Comment:
**major:** This sample sits after `runPendingTableServicesOperationsAndRefreshTimeline`, which rethrows (L1623) when a pending MDT compaction fails, so in the "compaction keeps failing" state this line is never reached again and the gauge freezes at its last value while delta commits keep landing. That is silent only with `hoodie.metadata.write.fail.on.table.service.failures=false`, but that is the deployment relying on a metric. Could we sample in the `finally` block next to `TABLE_SERVICE_EXECUTION_DURATION`, from `metadataMetaClient.getActiveTimeline()`, so the failure path reports too?
<details><summary>trace</summary>
`compactIfNecessary` -> `writeClient.compact` throws -> L1667 `COMPACTION_FAILURES++`, rethrow -> compaction instant left pending. Next cycle: L1539 -> `filterPendingCompactionTimeline().countInstants() > 0` -> `runAnyPendingCompactions()` throws -> L1621 `PENDING_COMPACTIONS_FAILURES++`, L1623 rethrow -> outer catch at L1570 -> L1556 skipped. Repeats every cycle until the pending compaction succeeds.
</details>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1543,8 +1543,20 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
.filterCompletedInstants()
.lastInstant();
if (!lastInstant.isPresent()) {
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(0L));
return;
}
+ // Report how far the metadata table has drifted from its last compaction, so that a stalled
+ // compaction is visible before the log to base file ratio degrades read performance. This is the
+ // backlog as it stands before the compaction below runs, so on a healthy table it peaks at
+ // hoodie.metadata.compact.max.delta.commits on the cycle that compacts and drops on the next one.
+ // It counts the same completed instants that ScheduleCompactionActionExecutor compares against
+ // that threshold, and is not re-sampled afterwards because doing so would need another timeline
+ // reload on the write path.
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(
+ CompactionUtils.getCompletedDeltaCommitsSinceLatestCompaction(activeTimeline)
+ .map(deltaCommitsInfo -> (long) deltaCommitsInfo.getLeft().countInstants())
+ .orElse(0L)));
Review Comment:
**nit:** Feel free to ignore. `.orElse(0L)` is unreachable: L1544 already returned unless a completed delta commit exists, and `getDeltaCommitsSinceLatestCompaction` returns empty only when the delta-commit timeline is empty (`CompactionUtils.java:329-335`). Could we use `.get()` here, or keep the fallback with a one-line "defensive" comment?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +85,22 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges
+ // show the current shape of the metadata table, but not whether compaction is keeping up with it.
+ // Completed delta commits on the metadata table since the last completed compaction, sampled before
+ // table services run. On a healthy table this sawtooths up to hoodie.metadata.compact.max.delta.commits
+ // and falls back after each compaction, so alert on a multiple of that config rather than on the
+ // config value itself. A value that climbs past the peak and keeps going means metadata table
+ // compaction is not being scheduled or is failing.
+ public static final String STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION = "deltaCommitsSinceLastCompaction";
Review Comment:
**major:** None of the four new names has a dot, so the CloudWatch reporter drops them: `CloudWatchReporter.stageMetricDatum` (c63c9bfa79fd, #19476) skips dot-less names, citing open #19507, which names `HoodieMetadataMetrics` as the cause. HUDI-9068 (100e9ac47590) prefixed `TABLE_SERVICE_EXECUTION_*` with the MDT table name in `performTableServices` for exactly this reason. Could we prefix these four the same way (`writeClient.getConfig().getTableName()` there, `metaClient.getTableConfig().getTableName()` in `updateSizeMetrics`)?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
+ long totalLogFiles = sumStat(stats, metadataPartitions, STAT_COUNT_LOG_FILES);
+ setMetric(STAT_TOTAL_BASE_FILE_COUNT, totalBaseFiles);
+ setMetric(STAT_TOTAL_LOG_FILE_COUNT, totalLogFiles);
+ setMetric(STAT_LOG_TO_BASE_FILE_RATIO_PERCENT, logToBaseFileRatioPercent(totalBaseFiles, totalLogFiles));
Review Comment:
**major:** `updateSizeMetrics` has no test anywhere (`grep -rn updateSizeMetrics` hits only two main-source lines), and the three new gauges are asserted only as pure helpers, so swapping the two `setMetric` arguments on L186-187 passes every test. Both existing `testMetadataMetrics` (`TestHoodieBackedMetadata.java:3812-3815`, `TestJavaHoodieBackedMetadata.java:2616-2623`) already assert the per-partition gauges from this same call. Could we add `containsKey` for the four new names there, rather than in a new file?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
+ long totalLogFiles = sumStat(stats, metadataPartitions, STAT_COUNT_LOG_FILES);
+ setMetric(STAT_TOTAL_BASE_FILE_COUNT, totalBaseFiles);
+ setMetric(STAT_TOTAL_LOG_FILE_COUNT, totalLogFiles);
+ setMetric(STAT_LOG_TO_BASE_FILE_RATIO_PERCENT, logToBaseFileRatioPercent(totalBaseFiles, totalLogFiles));
+ }
+
+ /**
+ * Sums a per-partition stat, keyed as {@code <partition>.<statName>}, over the enabled metadata partitions.
+ */
+ @VisibleForTesting
+ static long sumStat(Map<String, String> stats, Set<String> metadataPartitions, String statName) {
+ return metadataPartitions.stream()
+ .mapToLong(partition -> Long.parseLong(stats.getOrDefault(partition + "." + statName, "0")))
+ .sum();
+ }
+
+ /**
+ * Log files per base file expressed as a percentage, so that it can be reported through a long valued gauge.
+ *
+ * <p>Returns 0 when there is no base file to divide by. That case is not necessarily healthy: a metadata
+ * table holding log files but no base file has never been compacted. Read this gauge together with
+ * {@link #STAT_TOTAL_BASE_FILE_COUNT} and {@link #STAT_TOTAL_LOG_FILE_COUNT} rather than on its own.
+ */
+ @VisibleForTesting
+ static long logToBaseFileRatioPercent(long totalBaseFiles, long totalLogFiles) {
+ return totalBaseFiles > 0 ? (totalLogFiles * 100L) / totalBaseFiles : 0L;
+ }
+
+ /**
+ * Reports the number of completed metadata table delta commits since the last completed compaction.
+ *
+ * <p>This counts the same instants that {@code ScheduleCompactionActionExecutor#needCompact} compares
+ * against the metadata table's delta commit threshold, which is configured by
+ * {@code hoodie.metadata.compact.max.delta.commits} (default 10) under the default {@code NUM_COMMITS}
+ * trigger strategy. Compaction fires once the count reaches that threshold, and the gauge is sampled
+ * before table services run, so on a healthy table it peaks at the threshold on the cycle where
+ * compaction fires and drops on the next one. Alert on a multiple of the threshold; a stalled
Review Comment:
**major:** "climbs past it without bound" does not hold for a never-scheduled stall. `CompactionUtils.getEarliestInstantToRetainForCompaction` (L451-457) pins the last compaction only while `numDeltaCommits < max`; at 10 or more the pin moves to a delta commit, MDT archival (`keep.min+1 / keep.max+1` = 21/31, `HoodieMetadataWriteUtils.java:293-296`) drops the compaction, and the helper's "no compaction" branch saturates the gauge at 21-31, indistinguishable from never-compacted. An alert at 3x-4x never fires. Could we reword this and the PR body to say it saturates near `hoodie.keep.min.commits`, and alert at about 2x the threshold?
<details><summary>why the other stall modes differ</summary>
A TSM-delegated stall leaves the compaction instant pending, which `TimelineArchiverV2` candidate #1 (earliest pending instant) pins, so that one does climb. A failing compaction never reaches the sample at all (see `HoodieBackedTableMetadataWriter.java:1556`). Only the never-scheduled case (e.g. `validateCompactionScheduling` false behind a long-running DT instant) hits the plateau; MDT candidate #5 pins at the DT's earliest active instant, which sits after the last MDT compaction, so it does not retain it either.
</details>
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
+ long totalLogFiles = sumStat(stats, metadataPartitions, STAT_COUNT_LOG_FILES);
+ setMetric(STAT_TOTAL_BASE_FILE_COUNT, totalBaseFiles);
+ setMetric(STAT_TOTAL_LOG_FILE_COUNT, totalLogFiles);
+ setMetric(STAT_LOG_TO_BASE_FILE_RATIO_PERCENT, logToBaseFileRatioPercent(totalBaseFiles, totalLogFiles));
+ }
+
+ /**
+ * Sums a per-partition stat, keyed as {@code <partition>.<statName>}, over the enabled metadata partitions.
+ */
+ @VisibleForTesting
+ static long sumStat(Map<String, String> stats, Set<String> metadataPartitions, String statName) {
+ return metadataPartitions.stream()
+ .mapToLong(partition -> Long.parseLong(stats.getOrDefault(partition + "." + statName, "0")))
+ .sum();
+ }
+
+ /**
+ * Log files per base file expressed as a percentage, so that it can be reported through a long valued gauge.
+ *
+ * <p>Returns 0 when there is no base file to divide by. That case is not necessarily healthy: a metadata
+ * table holding log files but no base file has never been compacted. Read this gauge together with
+ * {@link #STAT_TOTAL_BASE_FILE_COUNT} and {@link #STAT_TOTAL_LOG_FILE_COUNT} rather than on its own.
+ */
+ @VisibleForTesting
+ static long logToBaseFileRatioPercent(long totalBaseFiles, long totalLogFiles) {
+ return totalBaseFiles > 0 ? (totalLogFiles * 100L) / totalBaseFiles : 0L;
Review Comment:
**minor:** Not blocking. This ratio is file-group-weighted: `getStats` counts per latest file slice, `files` has one file group and `record_index` defaults to 10 (global: 10000), so RLI drowns the partition that actually hurts listings. Worked case: `files` 1 base + 20 logs, RLI 1000 + 1000 reports 101%, "about one log per base", while every partition listing merges 20 log files. Since `getStats` already has the per-partition ints, could we emit the max ratio across partitions instead of the sum ratio?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
+ long totalLogFiles = sumStat(stats, metadataPartitions, STAT_COUNT_LOG_FILES);
+ setMetric(STAT_TOTAL_BASE_FILE_COUNT, totalBaseFiles);
+ setMetric(STAT_TOTAL_LOG_FILE_COUNT, totalLogFiles);
+ setMetric(STAT_LOG_TO_BASE_FILE_RATIO_PERCENT, logToBaseFileRatioPercent(totalBaseFiles, totalLogFiles));
+ }
+
+ /**
+ * Sums a per-partition stat, keyed as {@code <partition>.<statName>}, over the enabled metadata partitions.
+ */
+ @VisibleForTesting
+ static long sumStat(Map<String, String> stats, Set<String> metadataPartitions, String statName) {
+ return metadataPartitions.stream()
+ .mapToLong(partition -> Long.parseLong(stats.getOrDefault(partition + "." + statName, "0")))
Review Comment:
**minor:** Not blocking. `getOrDefault(..., "0")` is unreachable: `getStats` writes all four keys for every member of the same `metadataPartitions` set (L132-158), so `sumStat` re-parses strings it just produced, and `missingPartitionStatsAreTreatedAsZero` plus the `record_index` half of `statsAreSummedAcrossEnabledPartitionsOnly` assert states production cannot produce. Could we sum the two ints inside `getStats` and drop `sumStat` and those two tests?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +85,22 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges
+ // show the current shape of the metadata table, but not whether compaction is keeping up with it.
+ // Completed delta commits on the metadata table since the last completed compaction, sampled before
+ // table services run. On a healthy table this sawtooths up to hoodie.metadata.compact.max.delta.commits
+ // and falls back after each compaction, so alert on a multiple of that config rather than on the
+ // config value itself. A value that climbs past the peak and keeps going means metadata table
+ // compaction is not being scheduled or is failing.
Review Comment:
**minor:** Not blocking. With `hoodie.metadata.streaming.write.enabled`, `performTableServices` is never called (`FlinkHoodieBackedTableMetadataWriter.java:145` and `HoodieFlinkTableServiceClient.java:209` both guard on `!isStreamingWriteEnabled()`), so this gauge is never registered on the deployment where MDT compaction runs in a separate pipeline and can stall unnoticed. Could we note that gap in this comment, or emit it from the Flink compaction pipeline as well?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +85,22 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges
Review Comment:
**nit:** Feel free to ignore. `STAT_LAST_COMPACTION_TIMESTAMP` (L79) has been declared and never emitted since 298808baaf77 (2020-12-30). Since this block adds a compaction-health gauge next to it, would it be worth either wiring it up (an age-since-compaction gauge would not freeze on the failure path noted at `HoodieBackedTableMetadataWriter.java:1556`) or deleting it?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -651,4 +651,83 @@ void testPerformTableServicesWithFailureHandling(
// Verify metrics are incremented when there's a failure
verify(metrics, times(1)).incrementMetric(HoodieMetadataMetrics.PENDING_COMPACTIONS_FAILURES, 1);
}
+
+ @Test
+ void performTableServicesReportsCompletedDeltaCommitsSinceLastCompaction() throws Exception {
+ // A completed compaction, then three completed delta commits after it, plus one still inflight.
+ // The gauge must count the three completed instants only, so that it stays comparable to
+ // hoodie.compact.inline.max.delta.commits, which the compaction trigger evaluates the same way.
+ List<HoodieInstant> instants = new ArrayList<>();
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "001", "0011"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, "002", "0021"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "003", "0031"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "004", "0041"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "005", "0051"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, "006"));
+
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(instants)).performTableServices(Option.empty(), true);
+
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(3L);
Review Comment:
**minor:** Not blocking. Every delta commit here is requested and completed on the same side of the compaction's requested time, so this passes identically under `findInstantsAfter(requestedTime)`; the completion-time semantics HUDI-2461 (61f35ebe423d) introduced are unpinned. No case reaches 0 through the main branch either. Could we add `createNewInstant(COMPLETED, DELTA_COMMIT_ACTION, "0015", "0035")` (requested before "002", completed after) expecting 4, and a compaction-last case expecting 0?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -651,4 +651,83 @@ void testPerformTableServicesWithFailureHandling(
// Verify metrics are incremented when there's a failure
verify(metrics, times(1)).incrementMetric(HoodieMetadataMetrics.PENDING_COMPACTIONS_FAILURES, 1);
}
+
+ @Test
+ void performTableServicesReportsCompletedDeltaCommitsSinceLastCompaction() throws Exception {
+ // A completed compaction, then three completed delta commits after it, plus one still inflight.
+ // The gauge must count the three completed instants only, so that it stays comparable to
+ // hoodie.compact.inline.max.delta.commits, which the compaction trigger evaluates the same way.
+ List<HoodieInstant> instants = new ArrayList<>();
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "001", "0011"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, "002", "0021"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "003", "0031"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "004", "0041"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "005", "0051"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, "006"));
+
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(instants)).performTableServices(Option.empty(), true);
+
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(3L);
+ }
+
+ @Test
+ void performTableServicesReportsAllDeltaCommitsWhenNeverCompacted() throws Exception {
Review Comment:
**minor:** Not blocking. This scenario is already asserted instant-by-instant in `TestCompactionUtils.testGetDeltaCommitsSinceLatestCompaction` (`hudi-hadoop-common/.../TestCompactionUtils.java:258`, `prepareTimeline` L692-705), and it adds no wiring coverage over the sibling above. Could we fold the three `performTableServicesReports*` tests into one `@ParameterizedTest` over `(instants, expectedCount)`, as this file does at L135 and L532, dropping this row?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -651,4 +651,83 @@ void testPerformTableServicesWithFailureHandling(
// Verify metrics are incremented when there's a failure
verify(metrics, times(1)).incrementMetric(HoodieMetadataMetrics.PENDING_COMPACTIONS_FAILURES, 1);
}
+
+ @Test
+ void performTableServicesReportsCompletedDeltaCommitsSinceLastCompaction() throws Exception {
+ // A completed compaction, then three completed delta commits after it, plus one still inflight.
+ // The gauge must count the three completed instants only, so that it stays comparable to
+ // hoodie.compact.inline.max.delta.commits, which the compaction trigger evaluates the same way.
+ List<HoodieInstant> instants = new ArrayList<>();
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "001", "0011"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, "002", "0021"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "003", "0031"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "004", "0041"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "005", "0051"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, "006"));
+
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(instants)).performTableServices(Option.empty(), true);
+
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(3L);
+ }
+
+ @Test
+ void performTableServicesReportsAllDeltaCommitsWhenNeverCompacted() throws Exception {
+ // No compaction has ever run, so every completed delta commit counts as backlog.
+ List<HoodieInstant> instants = new ArrayList<>();
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "001", "0011"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "002", "0021"));
+
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(instants)).performTableServices(Option.empty(), true);
+
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(2L);
+ }
+
+ @Test
+ void performTableServicesReportsZeroDeltaCommitsOnEmptyTimeline() throws Exception {
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(new ArrayList<>()))
+ .performTableServices(Option.empty(), true);
+
+ // No completed delta commit yet, so the gauge is reset rather than left at a stale value.
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(0L);
+ }
+
+ /**
+ * Builds a partially mocked writer whose {@code performTableServices} runs for real against the given
+ * metadata timeline.
+ */
+ private static HoodieBackedTableMetadataWriter writerForTableServices(HoodieMetadataMetrics metrics,
Review Comment:
**minor:** Not blocking. This is the same setup as `testPerformTableServicesWithFailureHandling` (L573-640): four mocks, the same six stubs, the same four field injections and `doCallRealMethod` on `performTableServices`, except the older test still hand-rolls reflection instead of `setField` (L518). Could that test be switched onto this helper so there is one setup path?
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -523,7 +523,7 @@ private static void setField(Object target, String name, Object value) throws Ex
}
@SuppressWarnings("deprecation")
- private HoodieActiveTimeline createMockTimeline(List<HoodieInstant> instants) {
+ private static HoodieActiveTimeline createMockTimeline(List<HoodieInstant> instants) {
Review Comment:
**nit:** Feel free to ignore. All four callers of `createMockTimeline` are instance `@Test` methods and `writerForTableServices` never calls it, so the `static` here is unneeded churn. Could we revert this line?
```suggestion
private HoodieActiveTimeline createMockTimeline(List<HoodieInstant> instants) {
```
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -651,4 +651,83 @@ void testPerformTableServicesWithFailureHandling(
// Verify metrics are incremented when there's a failure
verify(metrics, times(1)).incrementMetric(HoodieMetadataMetrics.PENDING_COMPACTIONS_FAILURES, 1);
}
+
+ @Test
+ void performTableServicesReportsCompletedDeltaCommitsSinceLastCompaction() throws Exception {
+ // A completed compaction, then three completed delta commits after it, plus one still inflight.
+ // The gauge must count the three completed instants only, so that it stays comparable to
+ // hoodie.compact.inline.max.delta.commits, which the compaction trigger evaluates the same way.
Review Comment:
**nit:** Feel free to ignore. This names `hoodie.compact.inline.max.delta.commits`, while the javadoc on `updateDeltaCommitsSinceLastCompaction` names `hoodie.metadata.compact.max.delta.commits` (`HoodieMetadataConfig.java:141`, mapped onto the inline knob for the MDT write config at `HoodieMetadataWriteUtils.java:301`). Could we use the metadata key here too?
```suggestion
// hoodie.metadata.compact.max.delta.commits, which the compaction trigger evaluates the same way.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5743709453
Reopening to retrigger CI — the only remaining failure (test-spark-client-and-hadoop-common) is an infra flake in the coverage-merge step ("No coverage reports found"), not a real test failure; every actual test in that job passed (146/146).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5664965884
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
}, {
"hash" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048",
"triggerID" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"triggerType" : "PUSH"
}, {
"hash" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* 9625882157b3236831467434f43bc9fe3be4cf1a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048)
* 6e2fcf0145352e8035fb38616a45473e243e58d3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5588287832
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5742521854
> ⚠️ 🤖 _This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying._
>
> Thanks for the follow-up revisions! This round consolidates the compaction-backlog gauge into a single `reportDeltaCommitsSinceLastCompaction` helper invoked from the `finally` block of `performTableServices`, reloading the MDT timeline under the `metrics.ifPresent` guard, and reworks the tests to share a `writerForTableServices` helper with real `ActiveTimelineV2` instances. I traced the `CompactionUtils` path, the exception handling around the reload, and the production callers, and the logic holds up for the empty-timeline, failure, and reloaded-timeline cases. No issues flagged from this automated pass — a Hudi committer or PMC member can take it from here for a final review.
>
> cc @yihua
Done — extracted a qualifiedMetricName helper and used it for all three names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5743221455
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
}, {
"hash" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048",
"triggerID" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"triggerType" : "PUSH"
}, {
"hash" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17095",
"triggerID" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"triggerType" : "PUSH"
}, {
"hash" : "2dac8d64704793280240ec0aa2b4f0b122d713f8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17209",
"triggerID" : "2dac8d64704793280240ec0aa2b4f0b122d713f8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* 2dac8d64704793280240ec0aa2b4f0b122d713f8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17209)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-6272862384
@yihua @voonhous this has been green and fully addressed since Sep 20 — every review thread closed, CI passing, no changes since. Could one of you take a look when you have a moment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5586719713
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3996410910
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +84,26 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges show
+ // the current shape of the metadata table, but not whether compaction is keeping up with it. This is
+ // the count of completed delta commits since the last completed compaction, counting the same instants
+ // that ScheduleCompactionActionExecutor compares against hoodie.metadata.compact.max.delta.commits.
+ //
+ // It is sampled after table services have run, so a healthy table reports close to zero once a
+ // compaction lands, and a table whose compaction is failing or is never scheduled keeps reporting a
+ // backlog at or above the threshold. Alert on it staying at or above the threshold across consecutive
+ // cycles rather than on a fixed multiple of it: the value does not grow without bound. Once the backlog
+ // reaches the threshold the archiver stops pinning the last compaction
+ // (CompactionUtils#getEarliestInstantToRetainForCompaction), MDT archival eventually removes it, and the
+ // count then plateaus near hoodie.keep.min.commits, which depends on the archival config as well.
+ //
+ // It is emitted from HoodieBackedTableMetadataWriter#performTableServices, so it covers table services
+ // driven through that path. Under hoodie.metadata.streaming.write.enabled, Flink runs MDT compaction in
+ // a separate pipeline that does not call it, and this gauge is not emitted there. Read it alongside
+ // TABLE_SERVICE_EXECUTION_STATUS and COMPACTION_FAILURES, which distinguish a failing compaction from
+ // table services having stopped running at all.
+ public static final String STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION = "deltaCommitsSinceLastCompaction";
+
private final transient MetricRegistry metricsRegistry;
Review Comment:
Agreed, and this one is the right read now. Renamed to `DELTA_COMMITS_SINCE_LAST_COMPACTION` / `delta_commits_since_last_compaction` in 9625882.
The camelCase `STAT_*` family is what `getStats` emits per partition; this gauge is emitted from `performTableServices` next to `TABLE_SERVICE_EXECUTION_*` and `COMPACTION_FAILURES`, so it belongs to the snake_case operational family. The earlier suggestion to align it with `STAT_COUNT_BASE_FILES` was from the revision where it sat with the size gauges, which have since been dropped.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +84,26 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges show
+ // the current shape of the metadata table, but not whether compaction is keeping up with it. This is
+ // the count of completed delta commits since the last completed compaction, counting the same instants
+ // that ScheduleCompactionActionExecutor compares against hoodie.metadata.compact.max.delta.commits.
+ //
+ // It is sampled after table services have run, so a healthy table reports close to zero once a
+ // compaction lands, and a table whose compaction is failing or is never scheduled keeps reporting a
+ // backlog at or above the threshold. Alert on it staying at or above the threshold across consecutive
+ // cycles rather than on a fixed multiple of it: the value does not grow without bound. Once the backlog
+ // reaches the threshold the archiver stops pinning the last compaction
+ // (CompactionUtils#getEarliestInstantToRetainForCompaction), MDT archival eventually removes it, and the
+ // count then plateaus near hoodie.keep.min.commits, which depends on the archival config as well.
+ //
+ // It is emitted from HoodieBackedTableMetadataWriter#performTableServices, so it covers table services
+ // driven through that path. Under hoodie.metadata.streaming.write.enabled, Flink runs MDT compaction in
+ // a separate pipeline that does not call it, and this gauge is not emitted there. Read it alongside
+ // TABLE_SERVICE_EXECUTION_STATUS and COMPACTION_FAILURES, which distinguish a failing compaction from
+ // table services having stopped running at all.
+ public static final String STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION = "deltaCommitsSinceLastCompaction";
+
private final transient MetricRegistry metricsRegistry;
private final transient Metrics metrics;
Review Comment:
Done in 9625882 — dropped the `STAT_` prefix and moved the value to snake_case to match the operational metrics at this call site. See the thread on the newer revision of this comment for the reasoning.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1570,13 +1570,47 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
String executionStatusMetricName = tableNameExists
? String.format("%s.%s", metadataTableName, HoodieMetadataMetrics.TABLE_SERVICE_EXECUTION_STATUS)
: HoodieMetadataMetrics.TABLE_SERVICE_EXECUTION_STATUS;
+ String deltaCommitsMetricName = tableNameExists
+ ? String.format("%s.%s", metadataTableName, HoodieMetadataMetrics.STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION)
+ : HoodieMetadataMetrics.STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION;
long timeSpent = metadataTableServicesTimer.endTimer();
metrics.ifPresent(m -> m.setMetric(executionDurationMetricName, timeSpent));
if (allTableServicesExecutedSuccessfullyOrSkipped) {
metrics.ifPresent(m -> m.setMetric(executionStatusMetricName, 1));
} else {
metrics.ifPresent(m -> m.setMetric(executionStatusMetricName, -1));
}
+ reportDeltaCommitsSinceLastCompaction(deltaCommitsMetricName);
+ }
+ }
+
+ /**
+ * Reports the metadata table's compaction backlog, so that a stalled compaction is visible before the
+ * growing log to base file ratio shows up as slower metadata lookups.
+ *
+ * <p>Sampled here rather than alongside the compaction above so that the failure paths report too: a
+ * pending compaction that keeps failing throws out of
+ * {@link #runPendingTableServicesOperationsAndRefreshTimeline} before compaction is even reached, which
+ * is exactly the state this gauge exists to surface.
+ *
+ * <p>The timeline is reloaded rather than reused. Callers pass {@code requiresTimelineRefresh=false}
+ * ({@code HoodieTableMetadataWriter#performTableServices}), and neither {@code runAnyPendingCompactions}
+ * nor {@code writeClient.compact} refreshes this meta client - both go through their own - so the cached
+ * timeline can predate both the failure and the compaction being reported on.
+ */
+ private void reportDeltaCommitsSinceLastCompaction(String metricName) {
+ if (!metrics.isPresent()) {
+ return;
+ }
+ try {
Review Comment:
Done in 9625882 — folded it into `metrics.ifPresent(m -> ...)` with the try/catch inside the lambda, so the swallow-and-log behavior is unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5646455859
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
}, {
"hash" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* bde2784f650ada62b14f34af553a53a28ef946e6 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024)
* 9625882157b3236831467434f43bc9fe3be4cf1a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3991769875
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +85,22 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges
+ // show the current shape of the metadata table, but not whether compaction is keeping up with it.
+ // Completed delta commits on the metadata table since the last completed compaction, sampled before
+ // table services run. On a healthy table this sawtooths up to hoodie.metadata.compact.max.delta.commits
+ // and falls back after each compaction, so alert on a multiple of that config rather than on the
+ // config value itself. A value that climbs past the peak and keeps going means metadata table
+ // compaction is not being scheduled or is failing.
Review Comment:
Confirmed at both guards. I would rather not wire the Flink compaction pipeline into this PR, so I have scoped the comment to table services driven through `performTableServices` and called the streaming-write gap out explicitly for a follow-up.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
+ long totalLogFiles = sumStat(stats, metadataPartitions, STAT_COUNT_LOG_FILES);
+ setMetric(STAT_TOTAL_BASE_FILE_COUNT, totalBaseFiles);
+ setMetric(STAT_TOTAL_LOG_FILE_COUNT, totalLogFiles);
+ setMetric(STAT_LOG_TO_BASE_FILE_RATIO_PERCENT, logToBaseFileRatioPercent(totalBaseFiles, totalLogFiles));
+ }
+
+ /**
+ * Sums a per-partition stat, keyed as {@code <partition>.<statName>}, over the enabled metadata partitions.
+ */
+ @VisibleForTesting
+ static long sumStat(Map<String, String> stats, Set<String> metadataPartitions, String statName) {
+ return metadataPartitions.stream()
+ .mapToLong(partition -> Long.parseLong(stats.getOrDefault(partition + "." + statName, "0")))
Review Comment:
Right, and moot now: `sumStat` and both of those tests went away with the size gauges.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +85,22 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges
Review Comment:
Fair observation, but it has been dead since 2020 and either wiring or deleting it is a separate change from this one. An age-since-compaction gauge is a good idea, and I would rather propose it on its own than fold it in here.
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -651,4 +651,83 @@ void testPerformTableServicesWithFailureHandling(
// Verify metrics are incremented when there's a failure
verify(metrics, times(1)).incrementMetric(HoodieMetadataMetrics.PENDING_COMPACTIONS_FAILURES, 1);
}
+
+ @Test
+ void performTableServicesReportsCompletedDeltaCommitsSinceLastCompaction() throws Exception {
+ // A completed compaction, then three completed delta commits after it, plus one still inflight.
+ // The gauge must count the three completed instants only, so that it stays comparable to
+ // hoodie.compact.inline.max.delta.commits, which the compaction trigger evaluates the same way.
+ List<HoodieInstant> instants = new ArrayList<>();
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "001", "0011"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, "002", "0021"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "003", "0031"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "004", "0041"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "005", "0051"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, "006"));
+
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(instants)).performTableServices(Option.empty(), true);
+
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(3L);
Review Comment:
Agreed, the old cases passed under `findInstantsAfter` too. Added a `0015`/`0035` delta commit, requested before the compaction and completed after it, plus a compaction-last case expecting 0.
##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metadata/TestHoodieBackedTableMetadataWriter.java:
##########
@@ -651,4 +651,83 @@ void testPerformTableServicesWithFailureHandling(
// Verify metrics are incremented when there's a failure
verify(metrics, times(1)).incrementMetric(HoodieMetadataMetrics.PENDING_COMPACTIONS_FAILURES, 1);
}
+
+ @Test
+ void performTableServicesReportsCompletedDeltaCommitsSinceLastCompaction() throws Exception {
+ // A completed compaction, then three completed delta commits after it, plus one still inflight.
+ // The gauge must count the three completed instants only, so that it stays comparable to
+ // hoodie.compact.inline.max.delta.commits, which the compaction trigger evaluates the same way.
+ List<HoodieInstant> instants = new ArrayList<>();
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "001", "0011"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.COMMIT_ACTION, "002", "0021"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "003", "0031"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "004", "0041"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "005", "0051"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.INFLIGHT, HoodieTimeline.DELTA_COMMIT_ACTION, "006"));
+
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(instants)).performTableServices(Option.empty(), true);
+
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(3L);
+ }
+
+ @Test
+ void performTableServicesReportsAllDeltaCommitsWhenNeverCompacted() throws Exception {
+ // No compaction has ever run, so every completed delta commit counts as backlog.
+ List<HoodieInstant> instants = new ArrayList<>();
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "001", "0011"));
+ instants.add(INSTANT_GENERATOR.createNewInstant(
+ HoodieInstant.State.COMPLETED, HoodieTimeline.DELTA_COMMIT_ACTION, "002", "0021"));
+
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(instants)).performTableServices(Option.empty(), true);
+
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(2L);
+ }
+
+ @Test
+ void performTableServicesReportsZeroDeltaCommitsOnEmptyTimeline() throws Exception {
+ HoodieMetadataMetrics metrics = mock(HoodieMetadataMetrics.class);
+ writerForTableServices(metrics, createMockTimeline(new ArrayList<>()))
+ .performTableServices(Option.empty(), true);
+
+ // No completed delta commit yet, so the gauge is reset rather than left at a stale value.
+ verify(metrics, times(1)).updateDeltaCommitsSinceLastCompaction(0L);
+ }
+
+ /**
+ * Builds a partially mocked writer whose {@code performTableServices} runs for real against the given
+ * metadata timeline.
+ */
+ private static HoodieBackedTableMetadataWriter writerForTableServices(HoodieMetadataMetrics metrics,
Review Comment:
Done. `testPerformTableServicesWithFailureHandling` now goes through the same helper, with real timelines instead of deep stubs and `setField` instead of the hand-rolled reflection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5638807391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914)
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* bde2784f650ada62b14f34af553a53a28ef946e6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3996475487
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1570,16 +1570,55 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
String executionStatusMetricName = tableNameExists
? String.format("%s.%s", metadataTableName, HoodieMetadataMetrics.TABLE_SERVICE_EXECUTION_STATUS)
: HoodieMetadataMetrics.TABLE_SERVICE_EXECUTION_STATUS;
+ String deltaCommitsMetricName = tableNameExists
Review Comment:
🤖 nit: this is now the third copy of the `tableNameExists ? String.format("%s.%s", metadataTableName, X) : X` ternary in this block — might be worth a tiny `qualifiedMetricName(String)` helper so the three names read as one line each.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3961461366
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
Review Comment:
**minor:** Not blocking, but to make this concrete: `hoodie.metadata.enable.detailed.metrics` defaults to false (c6d707c6a8bd, #18803), added because `updateSizeMetrics` builds a `HoodieTableFileSystemView` on the driver. So by default only `deltaCommitsSinceLastCompaction` exists, the PR body's "four series per table" does not hold, and the ratio javadoc's "read together with" the two totals points at gauges that are off. The three aggregates are also derivable from the per-partition gauges this same call emits. Could we name the config in this comment block and in the PR body?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5665871079
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
}, {
"hash" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048",
"triggerID" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"triggerType" : "PUSH"
}, {
"hash" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17095",
"triggerID" : "6e2fcf0145352e8035fb38616a45473e243e58d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* 9625882157b3236831467434f43bc9fe3be4cf1a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048)
* 6e2fcf0145352e8035fb38616a45473e243e58d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17095)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5586628864
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5586540563
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5744262785
Retriggering again — last failure was a transient Maven Central 403 fetching os-maven-plugin during Build Project, unrelated to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli closed pull request #19868: feat(metadata): report metadata table compaction health metrics
URL: https://github.com/apache/hudi/pull/19868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3991768247
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1543,8 +1543,20 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
.filterCompletedInstants()
.lastInstant();
if (!lastInstant.isPresent()) {
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(0L));
return;
}
+ // Report how far the metadata table has drifted from its last compaction, so that a stalled
+ // compaction is visible before the log to base file ratio degrades read performance. This is the
+ // backlog as it stands before the compaction below runs, so on a healthy table it peaks at
+ // hoodie.metadata.compact.max.delta.commits on the cycle that compacts and drops on the next one.
+ // It counts the same completed instants that ScheduleCompactionActionExecutor compares against
+ // that threshold, and is not re-sampled afterwards because doing so would need another timeline
+ // reload on the write path.
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(
Review Comment:
Confirmed, and moved it into the `finally` block. One correction to the suggestion though: `getActiveTimeline()` there is not reliable either, since every caller passes `requiresTimelineRefresh=false` (`HoodieTableMetadataWriter.java:178`) and neither `runAnyPendingCompactions` nor `writeClient.compact` refreshes this meta client. Sampling off `reloadActiveTimeline()` instead, wrapped so a metric failure cannot mask the original exception.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1543,8 +1543,20 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
.filterCompletedInstants()
.lastInstant();
if (!lastInstant.isPresent()) {
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(0L));
return;
}
+ // Report how far the metadata table has drifted from its last compaction, so that a stalled
+ // compaction is visible before the log to base file ratio degrades read performance. This is the
+ // backlog as it stands before the compaction below runs, so on a healthy table it peaks at
+ // hoodie.metadata.compact.max.delta.commits on the cycle that compacts and drops on the next one.
+ // It counts the same completed instants that ScheduleCompactionActionExecutor compares against
+ // that threshold, and is not re-sampled afterwards because doing so would need another timeline
+ // reload on the write path.
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(
+ CompactionUtils.getCompletedDeltaCommitsSinceLatestCompaction(activeTimeline)
+ .map(deltaCommitsInfo -> (long) deltaCommitsInfo.getLeft().countInstants())
+ .orElse(0L)));
Review Comment:
Right as it stood, but moving the sample into `finally` drops the `lastInstant` guard from that path, so an empty delta-commit timeline now reaches it. Keeping `.orElse(0L)` as the real zero case.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
+ long totalLogFiles = sumStat(stats, metadataPartitions, STAT_COUNT_LOG_FILES);
+ setMetric(STAT_TOTAL_BASE_FILE_COUNT, totalBaseFiles);
+ setMetric(STAT_TOTAL_LOG_FILE_COUNT, totalLogFiles);
+ setMetric(STAT_LOG_TO_BASE_FILE_RATIO_PERCENT, logToBaseFileRatioPercent(totalBaseFiles, totalLogFiles));
+ }
+
+ /**
+ * Sums a per-partition stat, keyed as {@code <partition>.<statName>}, over the enabled metadata partitions.
+ */
+ @VisibleForTesting
+ static long sumStat(Map<String, String> stats, Set<String> metadataPartitions, String statName) {
+ return metadataPartitions.stream()
+ .mapToLong(partition -> Long.parseLong(stats.getOrDefault(partition + "." + statName, "0")))
+ .sum();
+ }
+
+ /**
+ * Log files per base file expressed as a percentage, so that it can be reported through a long valued gauge.
+ *
+ * <p>Returns 0 when there is no base file to divide by. That case is not necessarily healthy: a metadata
+ * table holding log files but no base file has never been compacted. Read this gauge together with
+ * {@link #STAT_TOTAL_BASE_FILE_COUNT} and {@link #STAT_TOTAL_LOG_FILE_COUNT} rather than on its own.
+ */
+ @VisibleForTesting
+ static long logToBaseFileRatioPercent(long totalBaseFiles, long totalLogFiles) {
+ return totalBaseFiles > 0 ? (totalLogFiles * 100L) / totalBaseFiles : 0L;
+ }
+
+ /**
+ * Reports the number of completed metadata table delta commits since the last completed compaction.
+ *
+ * <p>This counts the same instants that {@code ScheduleCompactionActionExecutor#needCompact} compares
+ * against the metadata table's delta commit threshold, which is configured by
+ * {@code hoodie.metadata.compact.max.delta.commits} (default 10) under the default {@code NUM_COMMITS}
+ * trigger strategy. Compaction fires once the count reaches that threshold, and the gauge is sampled
+ * before table services run, so on a healthy table it peaks at the threshold on the cycle where
+ * compaction fires and drops on the next one. Alert on a multiple of the threshold; a stalled
Review Comment:
Correct, reworded: it plateaus near `hoodie.keep.min.commits` once archival stops pinning the last compaction. I left the 2x out though, since the plateau depends on the archival config as much as on the compaction threshold. With the sample now post-service a healthy table returns to about zero each cycle, so the guidance is "stays at or above the threshold across consecutive cycles".
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +85,22 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges
+ // show the current shape of the metadata table, but not whether compaction is keeping up with it.
+ // Completed delta commits on the metadata table since the last completed compaction, sampled before
+ // table services run. On a healthy table this sawtooths up to hoodie.metadata.compact.max.delta.commits
+ // and falls back after each compaction, so alert on a multiple of that config rather than on the
+ // config value itself. A value that climbs past the peak and keeps going means metadata table
+ // compaction is not being scheduled or is failing.
+ public static final String STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION = "deltaCommitsSinceLastCompaction";
Review Comment:
Good catch, verified at `CloudWatchReporter.java:282`. Now prefixed with the MDT table name at the call site, under the same `StringUtils.nonEmpty` guard as `TABLE_SERVICE_EXECUTION_*`. Worth noting `Metrics#registerGauge` applies no prefix of its own, only `registerGauges` does.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
+ long totalLogFiles = sumStat(stats, metadataPartitions, STAT_COUNT_LOG_FILES);
+ setMetric(STAT_TOTAL_BASE_FILE_COUNT, totalBaseFiles);
+ setMetric(STAT_TOTAL_LOG_FILE_COUNT, totalLogFiles);
+ setMetric(STAT_LOG_TO_BASE_FILE_RATIO_PERCENT, logToBaseFileRatioPercent(totalBaseFiles, totalLogFiles));
Review Comment:
Dropped the three size gauges from this PR, so this no longer applies here. The gap is real and pre-existing though: `updateSizeMetrics` should be covered in `testMetadataMetrics` regardless, and I am happy to file that separately.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
+ long totalLogFiles = sumStat(stats, metadataPartitions, STAT_COUNT_LOG_FILES);
+ setMetric(STAT_TOTAL_BASE_FILE_COUNT, totalBaseFiles);
+ setMetric(STAT_TOTAL_LOG_FILE_COUNT, totalLogFiles);
+ setMetric(STAT_LOG_TO_BASE_FILE_RATIO_PERCENT, logToBaseFileRatioPercent(totalBaseFiles, totalLogFiles));
+ }
+
+ /**
+ * Sums a per-partition stat, keyed as {@code <partition>.<statName>}, over the enabled metadata partitions.
+ */
+ @VisibleForTesting
+ static long sumStat(Map<String, String> stats, Set<String> metadataPartitions, String statName) {
+ return metadataPartitions.stream()
+ .mapToLong(partition -> Long.parseLong(stats.getOrDefault(partition + "." + statName, "0")))
+ .sum();
+ }
+
+ /**
+ * Log files per base file expressed as a percentage, so that it can be reported through a long valued gauge.
+ *
+ * <p>Returns 0 when there is no base file to divide by. That case is not necessarily healthy: a metadata
+ * table holding log files but no base file has never been compacted. Read this gauge together with
+ * {@link #STAT_TOTAL_BASE_FILE_COUNT} and {@link #STAT_TOTAL_LOG_FILE_COUNT} rather than on its own.
+ */
+ @VisibleForTesting
+ static long logToBaseFileRatioPercent(long totalBaseFiles, long totalLogFiles) {
+ return totalBaseFiles > 0 ? (totalLogFiles * 100L) / totalBaseFiles : 0L;
Review Comment:
Your worked case is right, a sum ratio is dominated by whichever partition has the most file groups. Rather than switch to a max I dropped the ratio, since the per-partition counts are more actionable than either aggregate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3991772860
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1543,8 +1543,20 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
.filterCompletedInstants()
.lastInstant();
if (!lastInstant.isPresent()) {
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(0L));
Review Comment:
Collapsed to a single call site in the `finally` block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5638437364
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914)
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3960950185
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +85,22 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges
+ // show the current shape of the metadata table, but not whether compaction is keeping up with it.
+ // Completed delta commits on the metadata table since the last completed compaction, sampled before
+ // table services run. On a healthy table this sawtooths up to hoodie.metadata.compact.max.delta.commits
+ // and falls back after each compaction, so alert on a multiple of that config rather than on the
+ // config value itself. A value that climbs past the peak and keeps going means metadata table
+ // compaction is not being scheduled or is failing.
+ public static final String STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION = "deltaCommitsSinceLastCompaction";
+ // Base and log file counts summed across all enabled metadata partitions.
+ public static final String STAT_TOTAL_BASE_FILE_COUNT = "totalBaseFileCount";
+ public static final String STAT_TOTAL_LOG_FILE_COUNT = "totalLogFileCount";
+ // Log files per base file across the whole metadata table, as a percentage, since gauges are long
+ // valued: 100 means one log file per base file, 500 means five. A rising value means readers must
Review Comment:
🤖 nit: the existing per-partition stats are named `STAT_COUNT_BASE_FILES` / `STAT_COUNT_LOG_FILES` (COUNT first), but these new totals flip the order to `STAT_TOTAL_BASE_FILE_COUNT` (COUNT last). Might be worth aligning the naming pattern, e.g. `STAT_TOTAL_COUNT_BASE_FILES`, so the two families read consistently.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -164,6 +181,48 @@ public void updateSizeMetrics(HoodieTableMetaClient metaClient, HoodieBackedTabl
for (Map.Entry<String, String> e : stats.entrySet()) {
setMetric(e.getKey(), Long.parseLong(e.getValue()));
}
+ long totalBaseFiles = sumStat(stats, metadataPartitions, STAT_COUNT_BASE_FILES);
Review Comment:
🤖 Is it intended that these three gauges only show up when detailed metrics are on? The only caller of `updateSizeMetrics` (`HoodieBackedTableMetadataWriter` ~L1389) is guarded by `isDetailedMetricsEnabled()`, so `deltaCommitsSinceLastCompaction` will be emitted by default while `totalBaseFileCount`/`totalLogFileCount`/`logToBaseFileRatioPercent` won't — might be worth calling out in the description, since the docs here say to read them together.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1543,8 +1543,20 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
.filterCompletedInstants()
.lastInstant();
if (!lastInstant.isPresent()) {
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(0L));
Review Comment:
🤖 nit: there are now two separate `metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(...))` calls a few lines apart (one for the early-return zero case, one for the computed value). Consider factoring the delta-commit calculation into a small helper so the metric is updated from a single call site.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19868:
URL: https://github.com/apache/hudi/pull/19868#discussion_r3991861902
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1570,13 +1570,47 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
String executionStatusMetricName = tableNameExists
? String.format("%s.%s", metadataTableName, HoodieMetadataMetrics.TABLE_SERVICE_EXECUTION_STATUS)
: HoodieMetadataMetrics.TABLE_SERVICE_EXECUTION_STATUS;
+ String deltaCommitsMetricName = tableNameExists
+ ? String.format("%s.%s", metadataTableName, HoodieMetadataMetrics.STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION)
+ : HoodieMetadataMetrics.STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION;
long timeSpent = metadataTableServicesTimer.endTimer();
metrics.ifPresent(m -> m.setMetric(executionDurationMetricName, timeSpent));
if (allTableServicesExecutedSuccessfullyOrSkipped) {
metrics.ifPresent(m -> m.setMetric(executionStatusMetricName, 1));
} else {
metrics.ifPresent(m -> m.setMetric(executionStatusMetricName, -1));
}
+ reportDeltaCommitsSinceLastCompaction(deltaCommitsMetricName);
+ }
+ }
+
+ /**
+ * Reports the metadata table's compaction backlog, so that a stalled compaction is visible before the
+ * growing log to base file ratio shows up as slower metadata lookups.
+ *
+ * <p>Sampled here rather than alongside the compaction above so that the failure paths report too: a
+ * pending compaction that keeps failing throws out of
+ * {@link #runPendingTableServicesOperationsAndRefreshTimeline} before compaction is even reached, which
+ * is exactly the state this gauge exists to surface.
+ *
+ * <p>The timeline is reloaded rather than reused. Callers pass {@code requiresTimelineRefresh=false}
+ * ({@code HoodieTableMetadataWriter#performTableServices}), and neither {@code runAnyPendingCompactions}
+ * nor {@code writeClient.compact} refreshes this meta client - both go through their own - so the cached
+ * timeline can predate both the failure and the compaction being reported on.
+ */
+ private void reportDeltaCommitsSinceLastCompaction(String metricName) {
+ if (!metrics.isPresent()) {
+ return;
+ }
+ try {
Review Comment:
🤖 nit: the rest of this method uses `metrics.ifPresent(m -> ...)`; could you use the same style here instead of the explicit `isPresent()`/`get()` pair for consistency?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataMetrics.java:
##########
@@ -84,6 +84,26 @@ public class HoodieMetadataMetrics implements Serializable {
public static final String LOG_COMPACTION_FAILURES = "logcompaction_failures";
public static final String PENDING_COMPACTIONS_FAILURES = "pending_compactions_failures";
+ // Metadata table compaction health. The existing per-partition baseFileCount/logFileCount gauges show
+ // the current shape of the metadata table, but not whether compaction is keeping up with it. This is
+ // the count of completed delta commits since the last completed compaction, counting the same instants
+ // that ScheduleCompactionActionExecutor compares against hoodie.metadata.compact.max.delta.commits.
+ //
+ // It is sampled after table services have run, so a healthy table reports close to zero once a
+ // compaction lands, and a table whose compaction is failing or is never scheduled keeps reporting a
+ // backlog at or above the threshold. Alert on it staying at or above the threshold across consecutive
+ // cycles rather than on a fixed multiple of it: the value does not grow without bound. Once the backlog
+ // reaches the threshold the archiver stops pinning the last compaction
+ // (CompactionUtils#getEarliestInstantToRetainForCompaction), MDT archival eventually removes it, and the
+ // count then plateaus near hoodie.keep.min.commits, which depends on the archival config as well.
+ //
+ // It is emitted from HoodieBackedTableMetadataWriter#performTableServices, so it covers table services
+ // driven through that path. Under hoodie.metadata.streaming.write.enabled, Flink runs MDT compaction in
+ // a separate pipeline that does not call it, and this gauge is not emitted there. Read it alongside
+ // TABLE_SERVICE_EXECUTION_STATUS and COMPACTION_FAILURES, which distinguish a failing compaction from
+ // table services having stopped running at all.
+ public static final String STAT_DELTA_COMMITS_SINCE_LAST_COMPACTION = "deltaCommitsSinceLastCompaction";
+
private final transient MetricRegistry metricsRegistry;
private final transient Metrics metrics;
Review Comment:
🤖 nit: the neighboring constants in this block (TABLE_SERVICE_EXECUTION_STATUS, PENDING_COMPACTIONS_FAILURES, etc.) don't use a `STAT_` prefix — could you drop it here too for consistency?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1543,8 +1543,20 @@ public void performTableServices(Option<String> inFlightInstantTimestamp, boolea
.filterCompletedInstants()
.lastInstant();
if (!lastInstant.isPresent()) {
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(0L));
return;
}
+ // Report how far the metadata table has drifted from its last compaction, so that a stalled
+ // compaction is visible before the log to base file ratio degrades read performance. This is the
+ // backlog as it stands before the compaction below runs, so on a healthy table it peaks at
+ // hoodie.metadata.compact.max.delta.commits on the cycle that compacts and drops on the next one.
+ // It counts the same completed instants that ScheduleCompactionActionExecutor compares against
+ // that threshold, and is not re-sampled afterwards because doing so would need another timeline
+ // reload on the write path.
+ metrics.ifPresent(m -> m.updateDeltaCommitsSinceLastCompaction(
+ CompactionUtils.getCompletedDeltaCommitsSinceLatestCompaction(activeTimeline)
+ .map(deltaCommitsInfo -> (long) deltaCommitsInfo.getLeft().countInstants())
+ .orElse(0L)));
Review Comment:
🤖 Worth noting this nit flipped with the revision: now that the sample runs from `finally` with no early return, `getDeltaCommitsSinceLatestCompaction` does return empty for an MDT with no delta commits yet (`CompactionUtils.java:329-335`), so `.orElse(0L)` is reachable — the new "empty timeline" parameterized case asserts exactly that path. Switching to `.get()` here would throw on a fresh MDT, so the fallback should stay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5638884886
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914)
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* bde2784f650ada62b14f34af553a53a28ef946e6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli closed pull request #19868: feat(metadata): report metadata table compaction health metrics
URL: https://github.com/apache/hudi/pull/19868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub)" <gi...@apache.org>.
shangxinli closed pull request #19868: feat(metadata): report metadata table compaction health metrics
URL: https://github.com/apache/hudi/pull/19868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19868:
URL: https://github.com/apache/hudi/pull/19868#issuecomment-5647168169
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "48831c4267f422e038babd3bbfef008cd7229237",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "48831c4267f422e038babd3bbfef008cd7229237",
"triggerType" : "PUSH"
}, {
"hash" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16914",
"triggerID" : "a522fe0a9ff4aaf2cf9e85e52b472dd1e46e7f32",
"triggerType" : "PUSH"
}, {
"hash" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3cebc57beacf0e1e68156334e6e228137620a1ad",
"triggerType" : "PUSH"
}, {
"hash" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8745f3c53c413ca4a3915a8255eb54fe39987033",
"triggerType" : "PUSH"
}, {
"hash" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17024",
"triggerID" : "bde2784f650ada62b14f34af553a53a28ef946e6",
"triggerType" : "PUSH"
}, {
"hash" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048",
"triggerID" : "9625882157b3236831467434f43bc9fe3be4cf1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 48831c4267f422e038babd3bbfef008cd7229237 UNKNOWN
* 3cebc57beacf0e1e68156334e6e228137620a1ad UNKNOWN
* 8745f3c53c413ca4a3915a8255eb54fe39987033 UNKNOWN
* 9625882157b3236831467434f43bc9fe3be4cf1a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17048)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(metadata): report metadata table compaction health metrics [hudi]
Posted by "shangxinli (via GitHub) via commits" <co...@hudi.apache.org>.
shangxinli closed pull request #19868: feat(metadata): report metadata table compaction health metrics
URL: https://github.com/apache/hudi/pull/19868
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org