You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rangareddy (via GitHub)" <gi...@apache.org> on 2026/08/04 06:51:07 UTC

[I] HoodieMetadataMetrics registers metric names without the table prefix, so CloudWatch cannot derive a Table dimension [hudi]

rangareddy opened a new issue, #19507:
URL: https://github.com/apache/hudi/issues/19507

   ## Describe the problem
   
   `HoodieMetadataMetrics` registers gauges whose names do not carry the table prefix that every other Hudi metric carries, and in some cases carry no table name at all. This is the producer-side cause of the CloudWatch failures in #12182 / #13051.
   
   `HoodieMetadataMetrics` receives a `HoodieMetricsConfig` and discards it:
   
   ```java
   public HoodieMetadataMetrics(HoodieMetricsConfig metricsConfig, HoodieStorage storage, boolean detailedMetricsEnabled) {
     this.metrics = Metrics.getInstance(metricsConfig, storage);   // config not retained
   ```
   
   so `setMetric`/`incrementMetric` call `metrics.registerGauge(action, value)` directly, bypassing the `hoodie.metrics.reporter.metricsname.prefix` that `Metrics#registerGauges` applies. Two consequences:
   
   **1. Names with no dot at all** — nothing can derive a table from them:
   
   - `lookup_meta_index_bloom_filters_file_count` — `BaseTableMetadata.java:212`, on the normal bloom-index read path
   - `<partition>_bootstrap_error` — `HoodieBackedTableMetadataWriter.java:482`
   
   **2. Names with a dot but no table** — a consumer that derives a table from the first segment gets the wrong answer:
   
   - `<action>.count` / `<action>.totalDuration` — `HoodieMetadataMetrics.java:156-157`, giving `initialize.count`, `lookup_partitions.totalDuration`, ...
   - `<mdt_partition>.{baseFileCount,logFileCount,totalBaseFileSizeInBytes,totalLogFileSizeInBytes}` — `HoodieMetadataMetrics.java:137-140`, giving `files.baseFileCount`, `column_stats.logFileCount`, ...
   
   These are registered in practice — `TestJavaHoodieBackedMetadata` asserts the registry contains `initialize.count`, `files.baseFileCount` and friends.
   
   ## Why it matters beyond CloudWatch
   
   `AWSGlueCatalogSyncClient`... (no) — `CloudWatchReporter` derives its `Table` dimension by splitting the metric name on the first dot, so category 1 cannot be reported at all and category 2 is reported under a wrong `Table`. CloudWatch bills per unique metric name plus dimension set, so the wrong-dimension metrics are billed as distinct custom metrics.
   
   Other reporters are unaffected in behaviour but still emit inconsistent names.
   
   ## Suggested direction
   
   Retain the `HoodieMetricsConfig` in `HoodieMetadataMetrics` and apply `getMetricReporterMetricsNamePrefix()` in `setMetric` and `incrementMetric`. That fixes both categories, for every reporter at once, in one place.
   
   **This renames metrics**, so it breaks existing Graphite / Prometheus / JMX / Datadog dashboards and needs a release note — which is why #19476 deliberately did not do it. Prior fixes for the same symptom went producer-side (`1a5a9f7f03ec` [HUDI-4439] #6164, and `100e9ac47590` [HUDI-9068] #12873), so this is the direction with precedent.
   
   ## Context
   
   Raised from review on #19476, which makes `CloudWatchReporter` stop losing an entire batch when it meets one of these names. That is the "stop the bleeding" half; this issue is the cause.
   
   Credit to @voonhous for tracing the producer sites and the billing implication.
   
   ## Related
   
   - #19476
   - #12182, #13051
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org