You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2026/08/10 23:43:56 UTC

[PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

rahil-c opened a new pull request, #19575:
URL: https://github.com/apache/hudi/pull/19575

   ### Describe the issue this Pull Request addresses
   
   `HoodieMetadataMetrics` has declared three record-index lookup metrics for some time:
   
   ```java
   public static final String LOOKUP_RECORD_INDEX_TIME_STR = "lookup_record_index_time";
   public static final String LOOKUP_RECORD_INDEX_KEYS_COUNT_STR = "lookup_record_index_key_count";
   public static final String LOOKUP_RECORD_INDEX_KEYS_HITS_COUNT_STR = "lookup_record_index_key_hit_count";
   ```
   
   Nothing in the repository references any of them, and `HoodieBackedTableMetadata#readRecordIndexLocationsWithKeys` records why:
   
   ```java
   // TODO [HUDI-9544]: Metric does not work for rdd based API due to lazy evaluation.
   ```
   
   So with RLI enabled there is currently no way to tell how many keys were looked up, how many hit the index, or how many shards were read — which is often the most expensive phase of an upsert.
   
   The numbers already exist. `keysToLookup.size()` and the size of the returned map are local variables inside `RecordIndexFileGroupLookupFunction.call()`. But the function returns **only the hits** — a miss produces no output row — so the driver cannot recover the denominator from the resulting RDD at any price. Recovering `records_looked_up` or `shards_read` would each cost an extra Spark job over data already computed, and caller attribution is not reconstructible after the fact at all.
   
   Executors could always count; they had no way to report.
   
   Relates to HUDI-9544 and #19063.
   
   ### Summary and Changelog
   
   Four counters are emitted per RLI shard and published at each commit, so an operator can see the index hit rate, how much of the index was read, and whether the index is earning its keep.
   
   **`fix(metrics)`: harden `DistributedRegistry`** — these registries live in process-wide static maps that outlive the SparkContext, the write client and the table.
   - `set()` is last-writer-wins, which is neither commutative nor associative; inside an `AccumulatorV2` the driver merges executor copies in an unspecified order, so an executor-side `set()` is non-deterministic. Now rejected from inside a task. No production caller exists today.
   - After a SparkContext restart in the same JVM (shells, notebooks, Spark Connect) the cached registry stayed bound to the dead context and executor updates silently stopped arriving. `isRegistered()` cannot detect this — it consults a Spark-global weak-reference table that `SparkContext.stop()` never clears, and the static map pins the registry so it is never collected. The application id is now stamped at registration and compared on lookup; on mismatch the entry is evicted and a **fresh** instance built, since `AccumulatorV2.register()` throws if the accumulator already carries registration metadata. Evict-and-recreate is atomic, or racing callers leave two live accumulators for one metric name with only one reachable.
   - `HoodieWrapperFileSystem` counters leaked into the next write client for the same table; now cleared on create and on close.
   
   **`feat(metrics)`: record and publish the counters** — `records_looked_up`, `hits`, `misses`, `shards_read`, each tagged by caller so tag-location traffic is distinguishable from the read client's dedupe traffic. The registry is resolved on the driver and captured by the lookup closure, so it rides inside the closure and nothing is resolved by name on an executor. Only `increment`/`add` are used.
   
   They are drained **once**, at the commit boundary, in `BaseHoodieClient#updateExtraMetadata`. The single drain is load-bearing: both sinks consume the registry destructively — the drain clears it, and `Registry.getAllMetrics(flush=true, …)` clears it when a reporter scrapes — so draining twice would give commit metadata *or* the reporter, never both. One read fans out to the commit's extra metadata as `hoodie.rli.lookup.<caller>.<metric>` and to the metrics reporter as gauges, then clears.
   
   Since every engine write path reaches `commitStats`, this covers Spark DataSource, all four Spark SQL DML commands and StreamSync by construction. It also gives the DeltaStreamer path a reporting cadence it did not have: `registerHoodieCommonMetrics` is otherwise reachable only from `Metrics.flush()` (no production callers) and `Metrics.shutdownAllMetrics()` (called only by `HoodieSparkSqlWriter`), so a long-running streaming job previously published these only from the JVM shutdown hook.
   
   **`test(metrics)`: functional coverage** on all three write paths, against both the global and the partitioned record index — separate closures on separate code paths. Assertions read counters off the latest commit on the timeline. Invariant throughout: `hits + misses == records_looked_up == incoming keys`.
   
   Measured, all passing:
   
   | Path | Global RLI | Partitioned RLI |
   |---|---|---|
   | DataSource upsert (20 updates + 1 insert) | 20 / 1 / 21, 8 shards | 20 / 1 / 21, 3 shards |
   | SQL `UPDATE` (optimized writes off) | 60 / 0 / 60, 10 shards | 60 / 0 / 60, 3 shards |
   | SQL `MERGE INTO` | 25 / 0 / 25 | 25 / 0 / 25 |
   | DeltaStreamer `sync()` ×2 | 500 / 500 / 1000, 10 shards | 500 / 500 / 1000, 3 shards |
   
   ### Impact
   
   **New config** `hoodie.metrics.rli.lookup.enable`, default `true`. When off, the registry is never resolved and executors do no extra work.
   
   **Behaviour worth flagging for reviewers.** Spark SQL `UPDATE` and `DELETE` perform no record-index lookup at all — they set `_hoodie.spark.sql.writes.prepped=true` whenever `hoodie.spark.sql.optimized.writes.enable` is on (it defaults to `true`), and a prepped write already knows each record's location from the rows it just read. Reporting nothing there is correct rather than a gap, and there are tests pinning both sides so a future change to the prepped path cannot silently alter what operators see. `MERGE INTO` is not a prepped write and does report.
   
   **Timeline footprint.** The commit-metadata sink adds roughly eight short entries per commit on an RLI table, and commit metadata is retained permanently including archives. That is the main reason for the config gate, and a fair thing to argue about — see the open question below.
   
   **Gauge semantics.** The reporter sink publishes gauges, so each commit overwrites the previous value: the reported number is the last commit's, not a running total. `rate()` will not behave as one might expect on these.
   
   No format change, no public API break.
   
   ### Risk Level
   
   low
   
   Additive and gated. The registry changes are the only ones touching existing behaviour: the `set()` rejection has no production callers today, and the staleness eviction only fires where updates were already being silently dropped. Verified by 11 registry tests plus 18 functional tests across three write paths and both index variants.
   
   ### Documentation Update
   
   The new config `hoodie.metrics.rli.lookup.enable` carries a full description, including the timeline-retention caveat. Happy to add a website page describing the counters if reviewers think this warrants one.
   
   ### Open question for reviewers
   
   Is `true` the right default? Observability out of the box is the argument for; permanent per-commit timeline growth is the argument against. I lean toward defaulting to `false` for a release or two and flipping once it has baked, and would rather take that steer now than after it ships.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   
   https://claude.ai/code/session_01VZsqnoc1EKhEa7459mh6sK
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5248757052

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0fd521723cb8a9ce999369619fe454e5ca47b4c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207) 
   * c4fa5fad9b2a0ec0866cd13778d7288c22275494 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858490348


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",
+      "rli",
+      "lookup",
+      HoodieWriteConfig::isRecordIndexLookupMetricsEnabled);
+
+  private final String registryName;
+  private final String commitMetadataPrefix;
+  private final String metricAction;
+  private final String metricQualifier;
+  private final Predicate<HoodieWriteConfig> enabled;
+
+  ExecutorMetricRegistry(String registryName, String commitMetadataPrefix, String metricAction,
+                         String metricQualifier, Predicate<HoodieWriteConfig> enabled) {
+    this.registryName = registryName;
+    this.commitMetadataPrefix = commitMetadataPrefix;
+    this.metricAction = metricAction;
+    this.metricQualifier = metricQualifier;
+    this.enabled = enabled;
+  }
+
+  /** The bare name emitting code passes to {@link Registry#getRegistry(String)}. */
+  @Override
+  public String registryName() {
+    return registryName;
+  }
+
+  @Override
+  public String commitMetadataPrefix() {
+    return commitMetadataPrefix;
+  }
+
+  @Override
+  public String metricAction() {
+    return metricAction;
+  }
+
+  @Override
+  public String metricQualifier() {
+    return metricQualifier;
+  }
+
+  /** Gating here, rather than in the drain, is what lets a new class of metric need no new config. */
+  @Override
+  public boolean isEnabled(HoodieWriteConfig config) {
+    return enabled.test(config);
+  }
+
+  /**
+   * Driver-side {@code REGISTRY_MAP} key. Table name alone is not an identity: two tables can share one
+   * and would then share a registry. Executors use the bare {@link #registryName()}.
+   */
+  @Override
+  public String scopedName(String basePath) {

Review Comment:
   Not sure i understand this comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3862767500


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {

Review Comment:
   The DataSource half holds only because `DefaultSource:194` calls `Metrics.shutdownAllMetrics()` after every write. `UpdateHoodieTableCommand:71`, `MergeIntoHoodieTableCommand:509` and `StreamSync` call `HoodieSparkSqlWriter.write` / the client directly and never shut metrics down between commits, so on those paths an abandoned attempt's counters still land on the next successful commit, and the flipped test runs the DataSource path only. Could we clear the group's registry at `startCommit` (your `dd4a48c`), which makes the guarantee path-independent, and run the failed-commit case once on the SQL class? We can follow this up in a second PR.
   



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {

Review Comment:
   The DataSource half holds only because `DefaultSource:194` calls `Metrics.shutdownAllMetrics()` after every write. `UpdateHoodieTableCommand:71`, `MergeIntoHoodieTableCommand:509` and `StreamSync` call `HoodieSparkSqlWriter.write` / the client directly and never shut metrics down between commits, so on those paths an abandoned attempt's counters still land on the next successful commit, and the flipped test runs the DataSource path only. Could we clear the group's registry at `startCommit` (your `dd4a48c`), which makes the guarantee path-independent, and run the failed-commit case once on the SQL class? We can follow this up in a second PR, if so, track it in an issue please.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863085631


##########
hudi-io/src/main/java/org/apache/hudi/common/metrics/Registry.java:
##########
@@ -190,6 +199,28 @@ static void setRegistries(Collection<Registry> registries) {
    */
   void set(String name, long value);
 
+  /**
+   * Subtract a set of counts previously read out of this registry, clamping every counter at zero.
+   *
+   * Used to hand a batch of counters over to a consumer that owns them from then on -- the commit-boundary
+   * drain for the record index lookup counters -- without discarding whatever arrived after they were read.
+   *
+   * Clamping is what distinguishes this from {@code add(name, -value)}. The registry can be emptied
+   * underneath a caller by an unrelated destructive scrape ({@link #getAllMetrics(boolean, boolean)} with
+   * {@code flush=true} clears every registry in the process), and an unbounded subtraction would then leave
+   * negative counters behind for good.
+   *
+   * The default is a best-effort read-modify-write. Implementations able to do this atomically should
+   * override it, and should drop counters that reach zero rather than leaving them at zero, so a registry
+   * nobody is writing to reads as empty.
+   *
+   * @param counts the counts to release, as returned by {@link #getAllCounts(boolean)}.
+   */
+  default void release(Map<String, Long> counts) {

Review Comment:
   Tracked on #19759 (section B) with the rest of the hudi-io lookup surface; not blocking here.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDReadClient.java:
##########
@@ -210,8 +212,16 @@ public JavaRDD<HoodieRecord<T>> filterExists(JavaRDD<HoodieRecord<T>> hoodieReco
    * @return Tagged RDD of Hoodie records
    */
   public JavaRDD<HoodieRecord<T>> tagLocation(JavaRDD<HoodieRecord<T>> hoodieRecords) throws HoodieIndexException {
-    return HoodieJavaRDD.getJavaRDD(
-        index.tagLocation(HoodieJavaRDD.of(hoodieRecords), context, hoodieTable));
+    // Lookups driven from the read client are dedupe traffic, not tag-location traffic. Label them so
+    // the two are distinguishable in the reported counters. Driver-side only: the label is
+    // captured when the lookup closure is built.
+    String previousCaller = RecordIndexLookupMetrics.setCaller(RecordIndexMetricNames.CALLER_DEDUPE);

Review Comment:
   Tracked on #19759 (section C) for restoration with coverage.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3867084520


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -142,9 +146,19 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
     return Option.empty();
   }
 
+  /**
+   * Carries thread-local task state onto Hudi's writer pools. The metrics binding travels with the
+   * {@link TaskContext} for the same reason that does: {@code BoundedInMemoryExecutor} and
+   * {@code DisruptorExecutor} run on threads the task did not create. No unbind, since these pools are
+   * created and shut down per operation.
+   */
   @Override
   public Runnable getPreExecuteRunnable() {
     final TaskContext taskContext = TaskContext.get();
-    return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+    final Map<String, Registry> metricsBinding = ExecutorMetricsContext.capture();

Review Comment:
   non-blocking: Tracing the three `bind` call sites, this one can only ever capture an empty map — the two lookup functions bind and unbind within a single `call()` (and `unbind(null)` does `BOUND.remove()`), and `getPreExecuteRunnable` is only reached from the write stage (`HoodieMergeHelper`, `SparkLazyInsertIterable`, the bootstrap handlers, `SingleSparkJobConsistentHashingExecutionStrategy`), never from inside a lookup. So the net effect is that `isBound()` becomes true on every writer pool thread with nothing bound, which is exactly the case where `Registry.getRegistry(name)` now returns `NoOpRegistry` instead of the process-wide `LocalRegistry`. Would it be simpler to drop this hunk until something below the write API actually emits, rather than carrying the flip for a binding that is always empty?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,47 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistry(String tableName, String registryName) {
+    String prefixedName = tableName.isEmpty() ? registryName : tableName + "." + registryName;
+    DISTRIBUTED_REGISTRY_MAP.remove(prefixedName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+    // setRegistries also indexes under the empty table name, so evicting only the table-scoped key leaves
+    // the accumulator reachable under ::<table>.<registry>.
+    Registry.REGISTRY_MAP.remove(Registry.makeKey("", prefixedName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {
     final String prefixedName = tableName.isEmpty() ? registryName : tableName + "." + registryName;
-    return DISTRIBUTED_REGISTRY_MAP.computeIfAbsent(prefixedName, key -> {
+    // Both maps are process-wide statics that outlive any SparkContext, so the staleness check and the
+    // recreation have to be atomic: otherwise one caller can evict the registry another caller just
+    // created, leaving two live accumulators for one metric name while reporting only ever reads the
+    // one still in the map.
+    return DISTRIBUTED_REGISTRY_MAP.compute(prefixedName, (key, cached) -> {
+      if (cached instanceof DistributedRegistry && ((DistributedRegistry) cached).isRegisteredWith(javaSparkContext)) {
+        return cached;
+      }
+      // Nothing usable cached, or the cached accumulator is bound to a SparkContext that is no longer
+      // live (a restart in the same JVM: shells, notebooks, Spark Connect). Drop the shared-map entry
+      // first, since getRegistryOfClass() would otherwise hand back that same stale instance.
+      final String sharedKey = Registry.makeKey(tableName, registryName);
+      Registry.REGISTRY_MAP.remove(sharedKey);

Review Comment:
   non-blocking: `removeMetricRegistry` above drops all three keys and its comment explains why the `::<table>.<registry>` one matters, but this eviction path only removes `makeKey(tableName, registryName)`. In local mode `setRegistries` puts the registry into the driver's own `REGISTRY_MAP` under `makeKey("", prefixedName)`, so after a SparkContext restart that entry still points at the dead accumulator while the new one is registered under the table-scoped key — and since both report the same prefixed names from `getAllMetrics`, which value survives depends on map iteration order. Worth adding the same `Registry.REGISTRY_MAP.remove(Registry.makeKey("", prefixedName))` here?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.NoOpRegistry;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricRegistry metricRegistry : ExecutorMetricRegistry.values()) {
+      if (!metricRegistry.isEnabled(config)) {

Review Comment:
   Since the "follows `hoodie.metrics.on`" behaviour comes from the infer function, an explicit `hoodie.metrics.rli.lookup.enable=true` with `hoodie.metrics.on=false` still registers the accumulator and counts on every task, but `HoodieMetrics.metrics` is null so `publishToReporter` returns at its null guard and the counters are released unread each commit. Would it be worth adding `config.isMetricsOn()` to the gate here, so the Impact section's "nothing is collected and no accumulator is created" holds for the explicit setting too?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3832154495


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {
+    for (Drained drained : counters.drained) {
+      publishToReporter(drained, hoodieMetrics);
+      drained.registry.release(drained.counts);
+    }
+  }
+
+  /** Gauges, so each commit overwrites the previous value rather than accumulating. */
+  private static void publishToReporter(Drained drained, HoodieMetrics hoodieMetrics) {
+    if (hoodieMetrics == null || hoodieMetrics.getMetrics() == null) {
+      return;
+    }
+    String prefix = hoodieMetrics.getMetricsName(
+        drained.metricRegistry.metricAction(), drained.metricRegistry.metricQualifier());
+    hoodieMetrics.getMetrics().registerGauges(drained.counts, Option.ofNullable(prefix));
+  }
+
+  private static final class Drained {
+    private final ExecutorMetricGroup metricRegistry;
+    private final Registry registry;
+    private final Map<String, Long> counts;
+
+    private Drained(ExecutorMetricGroup metricRegistry, Registry registry, Map<String, Long> counts) {

Review Comment:
   🤖 nit: could you rename `metricRegistry` to `metricGroup` here (and in the matching constructor parameter and loop variables in `snapshotIntoCommitMetadata` and `RecordIndexLookupMetrics.resolveBundle`)? The type is `ExecutorMetricGroup`, not a `Registry`, and having `metricRegistry` and `registry` as siblings in the same class requires a double-take to tell them apart.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3867788654


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry {

Review Comment:
   Any Spark-specific classes should not live in `hudi-client-common` module.  Could you move these to `hudi-spark-client` module?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -275,6 +276,10 @@ public boolean commitStats(String instantTime, TableWriteStats tableWriteStats,
       }
       commit(table, commitActionType, instantTime, metadata, tableWriteStats, skipStreamingWritesToMetadataTable);
       log.info("Committed {}", instantTime);
+      // The commit landed, so the executor counters it accounts for can be reported and released. A
+      // commit that never lands publishes nothing; its counters are not carried forward either, because
+      // Metrics.shutdown() clears every registry after each write on the DataSource path.
+      ExecutorMetrics.publishAndRelease(config, metrics);

Review Comment:
   Could you consider adding an abstraction or interface for the publishing, instead of adding `ExecutorMetrics` here which is Spark-specific?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5444695963

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * fa7936408666ceed9010ee10a19b38fa04bec224 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5260291269

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c4fa5fad9b2a0ec0866cd13778d7288c22275494 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211) 
   * 5d3fa9e8ab489adcd8b266911cfaee7672ab8846 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858579293


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/RecordIndexMetricNames.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+/**
+ * Counter names for the record index lookup phase. Collection itself is generic -- see
+ * {@link ExecutorMetricRegistry} and {@link ExecutorMetrics}.
+ */
+public class RecordIndexMetricNames {
+
+  /** Scoping, prefix and reporter naming all live on the enum entry. */
+  public static final String REGISTRY_NAME = ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.registryName();
+
+  public static final String COMMIT_METADATA_PREFIX =
+      ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.commitMetadataPrefix();
+
+  // Counters are tagged by caller so dedupe traffic is distinguishable from tag-location traffic.
+  public static final String CALLER_TAG_LOCATION = "tag";
+  public static final String CALLER_DEDUPE = "dedupe";
+
+  public static final String KEY_COUNT = "lookup_record_index_key_count";
+  public static final String KEY_HIT_COUNT = "lookup_record_index_key_hit_count";
+  public static final String KEY_MISS_COUNT = "lookup_record_index_key_miss_count";
+  public static final String SHARDS_READ = "lookup_record_index_shards_read";
+  /** Wall-clock spent in the shard read, summed across shards. Revives the third dead upstream constant,
+   * {@code HoodieMetadataMetrics.LOOKUP_RECORD_INDEX_TIME_STR}. */
+  public static final String LOOKUP_TIME = "lookup_record_index_time";

Review Comment:
   I need to check to see if these main the same if so will consolidate



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5419713141

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e62a131d0a4821a474fc78f037df1c82fcc1c559 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555) 
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * 221b2b5bfec5eb6548462bd91fe85b3502f52637 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556) 
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859019502


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la
+ */
+abstract class RliLookupMetricsTestBase extends RecordLevelIndexTestBase {

Review Comment:
   Partly taken. Deleted outright: `TestRliLookupMetricsReporting` (reporting is now the only sink, so every remaining test asserts through a reporter), `TestExecutorMetricsGenericity` (the abstraction is gone), and the Spark SQL class (that path does not report -- apache/hudi#19740).
   
   Not taking the rest yet. The MOR twins and the multi-table class are the only coverage of the two things most likely to break silently here -- table-type independence and per-table key separation -- and I would rather cut them once the registry lifetime settles, since that is what decides whether the digest is still load-bearing. Your point that different `TBL_NAME`s separate the keys on their own is fair; that makes the multi-table class weaker than intended rather than useless, and it is on the list.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",
+      "rli",
+      "lookup",
+      HoodieWriteConfig::isRecordIndexLookupMetricsEnabled);
+
+  private final String registryName;
+  private final String commitMetadataPrefix;
+  private final String metricAction;
+  private final String metricQualifier;
+  private final Predicate<HoodieWriteConfig> enabled;
+
+  ExecutorMetricRegistry(String registryName, String commitMetadataPrefix, String metricAction,
+                         String metricQualifier, Predicate<HoodieWriteConfig> enabled) {
+    this.registryName = registryName;
+    this.commitMetadataPrefix = commitMetadataPrefix;
+    this.metricAction = metricAction;
+    this.metricQualifier = metricQualifier;
+    this.enabled = enabled;
+  }
+
+  /** The bare name emitting code passes to {@link Registry#getRegistry(String)}. */
+  @Override
+  public String registryName() {
+    return registryName;
+  }
+
+  @Override
+  public String commitMetadataPrefix() {
+    return commitMetadataPrefix;
+  }
+
+  @Override
+  public String metricAction() {
+    return metricAction;
+  }
+
+  @Override
+  public String metricQualifier() {
+    return metricQualifier;
+  }
+
+  /** Gating here, rather than in the drain, is what lets a new class of metric need no new config. */
+  @Override
+  public boolean isEnabled(HoodieWriteConfig config) {
+    return enabled.test(config);
+  }
+
+  /**
+   * Driver-side {@code REGISTRY_MAP} key. Table name alone is not an identity: two tables can share one
+   * and would then share a registry. Executors use the bare {@link #registryName()}.
+   */
+  @Override
+  public String scopedName(String basePath) {

Review Comment:
   Real, and I hit it independently from the other direction while debugging: `Metrics.shutdown()` running `Registry.getAllMetrics(true, true)` is also why an abandoned commit's counters cannot survive to the retry on the DataSource path.
   
   In practice the republish does not fire today, because release-by-subtraction leaves the registry empty by the time teardown runs -- so it needs leftovers to be visible at all. Deferring the digest-free name / scrape exclusion to the registry-lifetime follow-up, since the digest only exists to disambiguate inside the process-wide map and both go away together.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/RecordIndexMetricNames.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+/**
+ * Counter names for the record index lookup phase. Collection itself is generic -- see
+ * {@link ExecutorMetricRegistry} and {@link ExecutorMetrics}.
+ */
+public class RecordIndexMetricNames {
+
+  /** Scoping, prefix and reporter naming all live on the enum entry. */
+  public static final String REGISTRY_NAME = ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.registryName();
+
+  public static final String COMMIT_METADATA_PREFIX =
+      ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.commitMetadataPrefix();
+
+  // Counters are tagged by caller so dedupe traffic is distinguishable from tag-location traffic.
+  public static final String CALLER_TAG_LOCATION = "tag";
+  public static final String CALLER_DEDUPE = "dedupe";
+
+  public static final String KEY_COUNT = "lookup_record_index_key_count";
+  public static final String KEY_HIT_COUNT = "lookup_record_index_key_hit_count";
+  public static final String KEY_MISS_COUNT = "lookup_record_index_key_miss_count";
+  public static final String SHARDS_READ = "lookup_record_index_shards_read";
+  /** Wall-clock spent in the shard read, summed across shards. Revives the third dead upstream constant,
+   * {@code HoodieMetadataMetrics.LOOKUP_RECORD_INDEX_TIME_STR}. */
+  public static final String LOOKUP_TIME = "lookup_record_index_time";

Review Comment:
   Fair, and I have documented the difference rather than dropped it. `index.lookup.duration` is driver wall-clock for the whole `tagLocation` including scheduling; `lookup_record_index_time` is the sum of per-shard executor time inside the index read. Dividing the latter by `shards_read` gives mean per-shard cost, and comparing the two shows how much of the lookup was actually index reading. That said, if you would rather this PR ship four counters and leave timing to the existing metric, I have no objection -- say the word.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/RecordIndexMetricNames.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+/**
+ * Counter names for the record index lookup phase. Collection itself is generic -- see
+ * {@link ExecutorMetricRegistry} and {@link ExecutorMetrics}.
+ */
+public class RecordIndexMetricNames {
+
+  /** Scoping, prefix and reporter naming all live on the enum entry. */
+  public static final String REGISTRY_NAME = ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.registryName();
+
+  public static final String COMMIT_METADATA_PREFIX =
+      ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.commitMetadataPrefix();
+
+  // Counters are tagged by caller so dedupe traffic is distinguishable from tag-location traffic.
+  public static final String CALLER_TAG_LOCATION = "tag";
+  public static final String CALLER_DEDUPE = "dedupe";
+
+  public static final String KEY_COUNT = "lookup_record_index_key_count";

Review Comment:
   (1) Taken as a doc fix: the value counts records, not distinct keys. Renaming the constant would change a metric name the requester specified, so the javadoc and config doc now say records explicitly instead.
   (2) The dead `HoodieMetadataMetrics.LOOKUP_RECORD_INDEX_*` constants and the `TODO [HUDI-9544]` are a good catch. Deferring to the same follow-up rather than touching `HoodieBackedTableMetadata` in a metrics PR, but it should be cleaned up when this lands.



##########
hudi-io/src/main/java/org/apache/hudi/common/metrics/Registry.java:
##########
@@ -190,6 +199,28 @@ static void setRegistries(Collection<Registry> registries) {
    */
   void set(String name, long value);
 
+  /**
+   * Subtract a set of counts previously read out of this registry, clamping every counter at zero.
+   *
+   * Used to hand a batch of counters over to a consumer that owns them from then on -- the commit-boundary
+   * drain for the record index lookup counters -- without discarding whatever arrived after they were read.
+   *
+   * Clamping is what distinguishes this from {@code add(name, -value)}. The registry can be emptied
+   * underneath a caller by an unrelated destructive scrape ({@link #getAllMetrics(boolean, boolean)} with
+   * {@code flush=true} clears every registry in the process), and an unbounded subtraction would then leave
+   * negative counters behind for good.
+   *
+   * The default is a best-effort read-modify-write. Implementations able to do this atomically should
+   * override it, and should drop counters that reach zero rather than leaving them at zero, so a registry
+   * nobody is writing to reads as empty.
+   *
+   * @param counts the counts to release, as returned by {@link #getAllCounts(boolean)}.
+   */
+  default void release(Map<String, Long> counts) {

Review Comment:
   Correct that the default is unreachable in production today -- the only `release` caller is the drain, whose registry is always a `DistributedRegistry`. Leaving it for now because the `LocalRegistry` override is what keeps the two implementations behaving the same way (it evicts counters reaching zero rather than setting them to zero, which the interface default cannot express), and dropping the default would make `release` abstract on a public hudi-io interface. Folding into the same hudi-io follow-up as the `Registry.getRegistry` contract change.



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metrics/TestDistributedRegistry.java:
##########
@@ -184,6 +187,131 @@ public void testAddMetricsParallel() {
     Assertions.assertEquals(finalExpectedSum, metricCounts.get(METRIC_1));
   }
 
+  @Test
+  public void testSetThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testSetOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: set() invoked on an executor must fail - it is non-commutative under accumulator merges.
+    // The UnsupportedOperationException thrown on the executor surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.set() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.set(METRIC_1, value);
+          return null;
+        }, 1));
+  }
+
+  @Test
+  public void testReleaseThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testReleaseOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: release() invoked on an executor must fail - clamping and eviction are order-dependent
+    // under accumulator merges. The UnsupportedOperationException surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.release() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.release(Collections.singletonMap(METRIC_1, (long) value));
+          return null;
+        }, 1));
+  }
+
+  /**
+   * Asserts the job failed because the executor-side guard fired, not for some unrelated reason such as a serialization error.
+   */
+  private static void assertFailsOnExecutorWith(String expectedMessage, Executable executable) {
+    SparkException thrown = Assertions.assertThrows(SparkException.class, executable);
+    StringBuilder chain = new StringBuilder();
+    for (Throwable t = thrown; t != null; t = t.getCause()) {
+      chain.append(t).append('\n');
+      if (t.getCause() == t) {
+        break;
+      }
+    }
+    Assertions.assertTrue(chain.toString().contains(expectedMessage),
+        "expected the executor-side guard to fail the job, got: " + chain);
+  }
+
+  @Test
+  public void testSetOnDriverSucceeds() {
+    // set() on the driver (no TaskContext) remains supported.
+    DistributedRegistry registry = new DistributedRegistry(REGISTRY_NAME + "_testSetOnDriver");
+    registry.set(METRIC_1, 42);
+    Assertions.assertEquals(42, registry.getAllCounts().get(METRIC_1));
+  }
+
+  @Test
+  public void testGetMetricRegistryReplacesNonDistributedRegistry() {

Review Comment:
   You are right that the test never enters the branch it names -- `getMetricRegistry` removes the seeded `LocalRegistry` before `getRegistryOfClass` runs, so `computeIfAbsent` creates a fresh `DistributedRegistry` and the `!(instanceof)` fallback stays uncovered. Deferring with the branch itself: if the registry becomes instance-scoped, the fallback goes away and so does the test. Noting it on the follow-up so it is not silently lost.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859031576


##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestRliMetricsOnStreamerPath.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.deltastreamer;
+
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The RLI lookup counters must reach commit metadata on the DeltaStreamer path, not only on the Spark DataSource path.
+ */
+@Tag("functional")
+public class TestRliMetricsOnStreamerPath extends HoodieDeltaStreamerTestBase {
+
+  /** Selects the global or partitioned record index. */
+  private static void enableRecordIndex(HoodieDeltaStreamer.Config cfg, boolean partitioned) {
+    cfg.configs.add(HoodieMetadataConfig.ENABLE.key() + "=true");
+    cfg.configs.add(HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key() + "=" + !partitioned);
+    cfg.configs.add(HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key() + "=" + partitioned);
+    cfg.configs.add(HoodieIndexConfig.INDEX_TYPE.key() + "="
+        + (partitioned ? "RECORD_LEVEL_INDEX" : "GLOBAL_RECORD_LEVEL_INDEX"));
+  }
+
+  private static Map<String, String> rliCountersOnLatestCommit(String tableBasePath) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HoodieTestUtils.getDefaultStorageConf())
+        .setBasePath(tableBasePath)
+        .build();
+    metaClient.reloadActiveTimeline();
+    HoodieInstant lastInstant = metaClient.getActiveTimeline()
+        .getCommitsTimeline().filterCompletedInstants().lastInstant().get();
+    HoodieCommitMetadata commitMetadata = metaClient.getActiveTimeline().readCommitMetadata(lastInstant);
+    Map<String, String> rli = new HashMap<>();
+    commitMetadata.getExtraMetadata().forEach((k, v) -> {
+      if (k.startsWith(RecordIndexMetricNames.COMMIT_METADATA_PREFIX)) {
+        rli.put(k, v);
+      }
+    });
+    return rli;
+  }
+
+  private static String tagKey(String metric) {
+    return RecordIndexMetricNames.COMMIT_METADATA_PREFIX
+        + RecordIndexMetricNames.key(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric);
+  }
+
+  @ParameterizedTest
+  @ValueSource(booleans = {false, true})
+  public void testRliCountersReachCommitMetadataOnStreamerPath(boolean partitioned) throws Exception {
+    String label = partitioned ? "partitioned" : "global";
+    String tableBasePath = basePath + "/test_rli_metrics_streamer_" + label;
+
+    // Sync 1 -- build the table and the record index.
+    HoodieDeltaStreamer.Config insertCfg =
+        TestHoodieDeltaStreamer.TestHelpers.makeConfig(tableBasePath, WriteOperationType.INSERT);
+    enableRecordIndex(insertCfg, partitioned);
+    new HoodieDeltaStreamer(insertCfg, jsc).sync();
+
+    // Sync 2 -- upsert, which tags incoming keys against the record index.
+    HoodieDeltaStreamer.Config upsertCfg =
+        TestHoodieDeltaStreamer.TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT);
+    enableRecordIndex(upsertCfg, partitioned);
+    new HoodieDeltaStreamer(upsertCfg, jsc).sync();
+
+    Map<String, String> counters = rliCountersOnLatestCommit(tableBasePath);
+
+    System.out.println("\n===== DeltaStreamer (" + label + " RLI) -- RLI counters on the commit =====");
+    if (counters.isEmpty()) {
+      System.out.println("  (none found)");
+    } else {
+      counters.entrySet().stream()
+          .sorted(Map.Entry.comparingByKey())
+          .forEach(e -> System.out.println(String.format("  %-52s %s", e.getKey(), e.getValue())));
+    }
+    System.out.println("==========================================================\n");
+
+    assertFalse(counters.isEmpty(),
+        "the commit-boundary drain must fire on the DeltaStreamer path; hudi-utilities never calls "
+            + "Metrics.shutdownAllMetrics, so nothing else would publish these");
+
+    String lookedUp = tagKey(RecordIndexMetricNames.KEY_COUNT);
+    assertTrue(counters.containsKey(lookedUp),
+        "tag-location traffic must be attributed on the streamer path too; got " + counters.keySet());
+
+    long records = Long.parseLong(counters.get(lookedUp));
+    long hits = Long.parseLong(counters.get(tagKey(RecordIndexMetricNames.KEY_HIT_COUNT)));
+    long misses = Long.parseLong(counters.get(tagKey(RecordIndexMetricNames.KEY_MISS_COUNT)));
+    assertTrue(records > 0, "the upsert sync looked up at least one key");
+    assertEquals(records, hits + misses, "hits + misses must account for every key looked up");

Review Comment:
   Correction to my reply above: I put the exact 1000/500/500 in, then could not verify it -- this test needs a Docker environment (`HoodieDeltaStreamerTestBase.initClass`) which I do not have locally, so it never reached the assertion. Rather than push numbers I had not confirmed, I backed off to asserting both `hits > 0` and `misses > 0` alongside the sum.
   
   That still fixes what you identified -- the sum alone cannot fail, these catch a run attributing everything to one side -- but it does not catch a doubled count the way exact values would. If you can confirm 1000/500/500 from a CI log, I am happy to pin them; you clearly have better visibility into that job than I do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5420981467

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 308c956e2a5895f99806e8d2d242f318cf670c42 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560) 
   * bf51bb77e90f1cfa0fed51257b7dd1cc2052a830 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5419348856

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8a71f71978edc6f83d37711a582e516b743aa7e7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468) 
   * e62a131d0a4821a474fc78f037df1c82fcc1c559 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555) 
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5419162328

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8a71f71978edc6f83d37711a582e516b743aa7e7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468) 
   * e62a131d0a4821a474fc78f037df1c82fcc1c559 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3871382137


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry {

Review Comment:
   Non-blocking abstraction suggestion: avoid generalizing before a second compatible metric shape exists.
   
   This enum currently has one constant, and `ExecutorMetrics` only knows how to publish flat counter maps. As a result, one feature is spread across `ExecutorMetricRegistry`, `ExecutorMetrics`, `RecordIndexMetricNames`, and `RecordIndexLookupMetrics`, while the next executor metric may need histograms or a different lifecycle anyway. Could `RecordIndexLookupMetrics` own its registry name, action, qualifier, counter names, and publish/release operation for now, with `SparkRDDWriteClient` calling it directly? A shared interface or enum can be extracted once a second compatible group provides concrete requirements.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5435167160

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610) 
   * 0ae8343d75734b1d9d89e04f1065337fc97faf9b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5365485845

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a061dbc0124d8ebc62defdf41d2d428842a6b16c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435) 
   * c3df340e3609cb1ad7f5f8abce5d207625887fa7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438) 
   * 457b1ac5c6ca7b2e46816e9c22d027bb498dbe41 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): make record index lookups observable via executor metrics [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5349327815

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f949227bee97d7b41d87d8c18ff73f6d15d7de17 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3840387481


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -255,6 +256,12 @@ public boolean commitStats(String instantTime, TableWriteStats tableWriteStats,
       return true;
     }
     extraMetadata = updateExtraMetadata(extraMetadata);
+    // The commit is the boundary for the record index lookup counters. Snapshot them into this commit's

Review Comment:
   can you make the comment here less verbose



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5247410166

   ## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19575?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `44.82759%` with `16 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 77.08%. Comparing base ([`4853b57`](https://app.codecov.io/gh/apache/hudi/commit/4853b57be208c47eaa0a976916e1130f77cb48ac?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f0fd521`](https://app.codecov.io/gh/apache/hudi/commit/f0fd521723cb8a9ce999369619fe454e5ca47b4c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19575?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...rg/apache/hudi/metrics/RecordIndexMetricNames.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetrics%2FRecordIndexMetricNames.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL21ldHJpY3MvUmVjb3JkSW5kZXhNZXRyaWNOYW1lcy5qYXZh) | 11.11% | [14 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #19575      +/-   ##
   ============================================
   - Coverage     77.09%   77.08%   -0.01%     
   - Complexity    32490    32491       +1     
   ============================================
     Files          2522     2523       +1     
     Lines        139112   139132      +20     
     Branches      16714    16716       +2     
   ============================================
   + Hits         107243   107247       +4     
   - Misses        24291    24305      +14     
   - Partials       7578     7580       +2     
   ```
   
   | [Components](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.25% <100.00%> (+<0.01%)` | :arrow_up: |
   | [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.67% <27.27%> (-0.05%)` | :arrow_down: |
   | [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.35% <ø> (ø)` | |
   | [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.59% <ø> (-0.01%)` | :arrow_down: |
   | [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.63% <ø> (ø)` | |
   | [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.32% <ø> (ø)` | |
   | [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.50% <ø> (ø)` | |
   | [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.11% <ø> (ø)` | |
   | [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.46% <ø> (ø)` | |
   | [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.44% <ø> (ø)` | |
   | [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `64.06% <ø> (ø)` | |
   | [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/19575/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
   
   | [Flag](https://app.codecov.io/gh/apache/hudi/pull/19575/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19575/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.73% <44.82%> (-0.01%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19575?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [.../java/org/apache/hudi/client/BaseHoodieClient.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2FBaseHoodieClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NsaWVudC9CYXNlSG9vZGllQ2xpZW50LmphdmE=) | `89.89% <100.00%> (+0.10%)` | :arrow_up: |
   | [...java/org/apache/hudi/config/HoodieWriteConfig.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fconfig%2FHoodieWriteConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2NvbmZpZy9Ib29kaWVXcml0ZUNvbmZpZy5qYXZh) | `92.04% <100.00%> (+<0.01%)` | :arrow_up: |
   | [...ava/org/apache/hudi/client/SparkRDDReadClient.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2FSparkRDDReadClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrUkREUmVhZENsaWVudC5qYXZh) | `88.13% <ø> (-0.39%)` | :arrow_down: |
   | [...va/org/apache/hudi/client/SparkRDDWriteClient.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2FSparkRDDWriteClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L1NwYXJrUkREV3JpdGVDbGllbnQuamF2YQ==) | `91.12% <ø> (ø)` | |
   | [...e/hudi/client/common/HoodieSparkEngineContext.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fclient%2Fcommon%2FHoodieSparkEngineContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY2xpZW50L2NvbW1vbi9Ib29kaWVTcGFya0VuZ2luZUNvbnRleHQuamF2YQ==) | `74.41% <ø> (-0.20%)` | :arrow_down: |
   | [...PartitionedRecordIndexFileGroupLookupFunction.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Findex%2FPartitionedRecordIndexFileGroupLookupFunction.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvUGFydGl0aW9uZWRSZWNvcmRJbmRleEZpbGVHcm91cExvb2t1cEZ1bmN0aW9uLmphdmE=) | `100.00% <ø> (ø)` | |
   | [...ndex/SparkMetadataTableGlobalRecordLevelIndex.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Findex%2FSparkMetadataTableGlobalRecordLevelIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvU3BhcmtNZXRhZGF0YVRhYmxlR2xvYmFsUmVjb3JkTGV2ZWxJbmRleC5qYXZh) | `90.90% <ø> (-0.32%)` | :arrow_down: |
   | [...hudi/index/SparkMetadataTableRecordLevelIndex.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Findex%2FSparkMetadataTableRecordLevelIndex.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvaW5kZXgvU3BhcmtNZXRhZGF0YVRhYmxlUmVjb3JkTGV2ZWxJbmRleC5qYXZh) | `95.00% <ø> (-0.24%)` | :arrow_down: |
   | [...a/org/apache/hudi/metrics/DistributedRegistry.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetrics%2FDistributedRegistry.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9EaXN0cmlidXRlZFJlZ2lzdHJ5LmphdmE=) | `93.93% <ø> (ø)` | |
   | [...g/apache/hudi/metrics/DistributedRegistryUtil.java](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fmetrics%2FDistributedRegistryUtil.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvbWV0cmljcy9EaXN0cmlidXRlZFJlZ2lzdHJ5VXRpbC5qYXZh) | `90.90% <ø> (ø)` | |
   | ... and [3 more](https://app.codecov.io/gh/apache/hudi/pull/19575?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5263920824

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f2ace685b8450e4466990371e11deba541aa992 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3778058069


##########
hudi-common/src/main/java/org/apache/hudi/common/config/metrics/HoodieMetricsConfig.java:
##########
@@ -109,6 +109,17 @@ public class HoodieMetricsConfig extends HoodieConfig {
       .sinceVersion("0.13.0")
       .withDocumentation("Enable metrics for locking infra. Useful when operating in multiwriter mode");
 
+  public static final ConfigProperty<Boolean> RLI_LOOKUP_METRICS_ENABLE = ConfigProperty
+      .key(METRIC_PREFIX + ".rli.lookup.enable")
+      .defaultValue(true)

Review Comment:
   🤖 Since this defaults to `true`, every RLI upsert commit will permanently stamp ~8 extra entries into commit metadata (retained through archival) for all existing RLI tables on upgrade, with no opt-in. You flagged this as the open question in the description — @yihua, do we want the default to be `false` given the permanent timeline footprint, or is default-on acceptable here?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/RecordIndexMetricNames.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * Counter names for the record level index lookup phase, and the commit-boundary drain that publishes
+ * them.
+ *
+ * <p>Lives in hudi-client-common rather than hudi-spark-client so the shared commit path can reach it.
+ * Because the drain happens on that shared path, it applies to every engine write path -- Spark
+ * DataSource, Spark SQL and DeltaStreamer -- without any of them knowing about it.
+ */
+public class RecordIndexMetricNames {
+
+  /** Base registry name; the key actually used is scoped per table instance, see {@link #registryName}. */
+  public static final String REGISTRY_NAME = "HoodieRecordIndexLookup";
+
+  /** Prefix under which counters are stamped into commit metadata. */

Review Comment:
   🤖 nit: the name `RecordIndexMetricNames` reads like a constants holder, but this class also owns the snapshot/publish/release drain logic and the `LookupCounters` type. Could you rename it to something that reflects that responsibility (e.g. `RecordIndexLookupCounters` or `...MetricsDrain`)? A caller seeing `RecordIndexMetricNames.snapshotIntoCommitMetadata(...)` doesn't expect behavior from a "Names" class.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5260326742

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c4fa5fad9b2a0ec0866cd13778d7288c22275494 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211) 
   * 5d3fa9e8ab489adcd8b266911cfaee7672ab8846 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): make record index lookups observable via executor metrics [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5362005597

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f949227bee97d7b41d87d8c18ff73f6d15d7de17 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391) 
   * 70968d47931e1e491260eccd0eadd85e12a83202 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5362757122

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 70968d47931e1e491260eccd0eadd85e12a83202 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426) 
   * 3d9cb9c3c5e0631abdad072dd0a6706ac42d4521 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3853278974


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {
+    for (Drained drained : counters.drained) {
+      publishToReporter(drained, hoodieMetrics);
+      drained.registry.release(drained.counts);
+    }
+  }
+
+  /** Gauges, so each commit overwrites the previous value rather than accumulating. */
+  private static void publishToReporter(Drained drained, HoodieMetrics hoodieMetrics) {
+    if (hoodieMetrics == null || hoodieMetrics.getMetrics() == null) {
+      return;
+    }
+    String prefix = hoodieMetrics.getMetricsName(
+        drained.metricRegistry.metricAction(), drained.metricRegistry.metricQualifier());
+    hoodieMetrics.getMetrics().registerGauges(drained.counts, Option.ofNullable(prefix));
+  }
+
+  private static final class Drained {

Review Comment:
   Two naming collisions worth avoiding. `ExecutorMetric*` now sits next to the unrelated `hoodie.metrics.executor.enable` mechanism (`DistributedRegistryUtil`), and "executor" already means the queue executors (`BoundedInMemoryExecutor`, `getPreExecuteRunnable`) in this codebase; `TaskMetric*` or `DistributedMetric*` would not collide. Separately, `rli` appears in no other config key (`hoodie.metadata.record.index.*`) and existing metric actions are spelled out (`archive`, `finalize`, `index`). Could the config key be `hoodie.metrics.record.index.lookup.enable` and the action `record_index`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5420470120

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * 221b2b5bfec5eb6548462bd91fe85b3502f52637 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556) 
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 308c956e2a5895f99806e8d2d242f318cf670c42 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560) 
   * bf51bb77e90f1cfa0fed51257b7dd1cc2052a830 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863088852


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metrics/TestDistributedRegistry.java:
##########
@@ -184,6 +187,131 @@ public void testAddMetricsParallel() {
     Assertions.assertEquals(finalExpectedSum, metricCounts.get(METRIC_1));
   }
 
+  @Test
+  public void testSetThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testSetOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: set() invoked on an executor must fail - it is non-commutative under accumulator merges.
+    // The UnsupportedOperationException thrown on the executor surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.set() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.set(METRIC_1, value);
+          return null;
+        }, 1));
+  }
+
+  @Test
+  public void testReleaseThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testReleaseOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: release() invoked on an executor must fail - clamping and eviction are order-dependent
+    // under accumulator merges. The UnsupportedOperationException surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.release() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.release(Collections.singletonMap(METRIC_1, (long) value));
+          return null;
+        }, 1));
+  }
+
+  /**
+   * Asserts the job failed because the executor-side guard fired, not for some unrelated reason such as a serialization error.
+   */
+  private static void assertFailsOnExecutorWith(String expectedMessage, Executable executable) {
+    SparkException thrown = Assertions.assertThrows(SparkException.class, executable);
+    StringBuilder chain = new StringBuilder();
+    for (Throwable t = thrown; t != null; t = t.getCause()) {
+      chain.append(t).append('\n');
+      if (t.getCause() == t) {
+        break;
+      }
+    }
+    Assertions.assertTrue(chain.toString().contains(expectedMessage),
+        "expected the executor-side guard to fail the job, got: " + chain);
+  }
+
+  @Test
+  public void testSetOnDriverSucceeds() {
+    // set() on the driver (no TaskContext) remains supported.
+    DistributedRegistry registry = new DistributedRegistry(REGISTRY_NAME + "_testSetOnDriver");
+    registry.set(METRIC_1, 42);
+    Assertions.assertEquals(42, registry.getAllCounts().get(METRIC_1));
+  }
+
+  @Test
+  public void testGetMetricRegistryReplacesNonDistributedRegistry() {

Review Comment:
   Could the test at least be renamed to what it proves (a stale `LocalRegistry` under the key is replaced by a fresh `DistributedRegistry`), so it does not claim coverage of the `!(instanceof)` fallback? The branch itself can wait for the lifetime change.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5426499673

   Blocked on @yihua 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863336511


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",
+      "rli",
+      "lookup",
+      HoodieWriteConfig::isRecordIndexLookupMetricsEnabled);
+
+  private final String registryName;
+  private final String commitMetadataPrefix;
+  private final String metricAction;
+  private final String metricQualifier;
+  private final Predicate<HoodieWriteConfig> enabled;
+
+  ExecutorMetricRegistry(String registryName, String commitMetadataPrefix, String metricAction,
+                         String metricQualifier, Predicate<HoodieWriteConfig> enabled) {
+    this.registryName = registryName;
+    this.commitMetadataPrefix = commitMetadataPrefix;
+    this.metricAction = metricAction;
+    this.metricQualifier = metricQualifier;
+    this.enabled = enabled;
+  }
+
+  /** The bare name emitting code passes to {@link Registry#getRegistry(String)}. */
+  @Override
+  public String registryName() {
+    return registryName;
+  }
+
+  @Override
+  public String commitMetadataPrefix() {
+    return commitMetadataPrefix;
+  }
+
+  @Override
+  public String metricAction() {
+    return metricAction;
+  }
+
+  @Override
+  public String metricQualifier() {
+    return metricQualifier;
+  }
+
+  /** Gating here, rather than in the drain, is what lets a new class of metric need no new config. */
+  @Override
+  public boolean isEnabled(HoodieWriteConfig config) {
+    return enabled.test(config);
+  }
+
+  /**
+   * Driver-side {@code REGISTRY_MAP} key. Table name alone is not an identity: two tables can share one
+   * and would then share a registry. Executors use the bare {@link #registryName()}.
+   */
+  @Override
+  public String scopedName(String basePath) {

Review Comment:
   Edited the PR body paragraph directly so it says what the scrape does (publishes whatever is left under `<prefix>.<table>.HoodieRecordIndexLookup.<12hex>.<metric>`, then clears); the fix itself is tracked on #19759. Resolving.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5443380715

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 0ae8343d75734b1d9d89e04f1065337fc97faf9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612) 
   * 47a7be9e0ce5d6aec98bb7cd582222a5d754462f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638) 
   * fa7936408666ceed9010ee10a19b38fa04bec224 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5444085268

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 47a7be9e0ce5d6aec98bb7cd582222a5d754462f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638) 
   * fa7936408666ceed9010ee10a19b38fa04bec224 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5447139654

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     }, {
       "hash" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644",
       "triggerID" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * fa7936408666ceed9010ee10a19b38fa04bec224 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639) 
   * 695f74f6364193e7869f83b7eab88e9b637b555c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5364936543

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a061dbc0124d8ebc62defdf41d2d428842a6b16c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5366363627

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 457b1ac5c6ca7b2e46816e9c22d027bb498dbe41 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5435628757

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 0ae8343d75734b1d9d89e04f1065337fc97faf9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5442682873

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 0ae8343d75734b1d9d89e04f1065337fc97faf9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612) 
   * 47a7be9e0ce5d6aec98bb7cd582222a5d754462f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859016150


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {

Review Comment:
   Taking the config half. `RLI_LOOKUP_METRICS_ENABLE` now defaults to `false` with `withInferFunction(cfg -> cfg.getBoolean(TURN_METRICS_ON))`, matching `LOCK_METRICS_ENABLE`. With the commit-metadata sink removed there is no longer any reason to collect when metrics are off -- there is nowhere to publish -- so this closes the opt-in gap rather than just narrowing it.
   
   The per-task accumulator cost is real and I could reproduce the unbounded growth: a single functional run leaves 40+ never-evicted `HoodieRecordIndexLookup.<digest>` entries in `REGISTRY_MAP`. Keeping the registry out of the engine-context captures is the right follow-up but it is the same change as the instance-scoped registry, so I am deferring both to the follow-up rather than reshaping the map in this PR.



##########
hudi-common/src/main/java/org/apache/hudi/common/config/metrics/HoodieMetricsConfig.java:
##########
@@ -109,6 +109,17 @@ public class HoodieMetricsConfig extends HoodieConfig {
       .sinceVersion("0.13.0")
       .withDocumentation("Enable metrics for locking infra. Useful when operating in multiwriter mode");
 
+  public static final ConfigProperty<Boolean> RLI_LOOKUP_METRICS_ENABLE = ConfigProperty
+      .key(METRIC_PREFIX + ".rli.lookup.enable")
+      .defaultValue(true)
+      .markAdvanced()
+      .sinceVersion("1.3.0")
+      .withDocumentation("Collect counters for the record level index lookup phase (records looked up, "

Review Comment:
   Fixed. The config doc now says Spark only and states that counters aggregate per table per JVM rather than per writer. It also no longer describes the commit-metadata sink, which has been removed.



##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestRliMetricsOnStreamerPath.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.deltastreamer;
+
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The RLI lookup counters must reach commit metadata on the DeltaStreamer path, not only on the Spark DataSource path.
+ */
+@Tag("functional")
+public class TestRliMetricsOnStreamerPath extends HoodieDeltaStreamerTestBase {
+
+  /** Selects the global or partitioned record index. */
+  private static void enableRecordIndex(HoodieDeltaStreamer.Config cfg, boolean partitioned) {
+    cfg.configs.add(HoodieMetadataConfig.ENABLE.key() + "=true");
+    cfg.configs.add(HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key() + "=" + !partitioned);
+    cfg.configs.add(HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key() + "=" + partitioned);
+    cfg.configs.add(HoodieIndexConfig.INDEX_TYPE.key() + "="
+        + (partitioned ? "RECORD_LEVEL_INDEX" : "GLOBAL_RECORD_LEVEL_INDEX"));
+  }
+
+  private static Map<String, String> rliCountersOnLatestCommit(String tableBasePath) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HoodieTestUtils.getDefaultStorageConf())
+        .setBasePath(tableBasePath)
+        .build();
+    metaClient.reloadActiveTimeline();
+    HoodieInstant lastInstant = metaClient.getActiveTimeline()
+        .getCommitsTimeline().filterCompletedInstants().lastInstant().get();
+    HoodieCommitMetadata commitMetadata = metaClient.getActiveTimeline().readCommitMetadata(lastInstant);
+    Map<String, String> rli = new HashMap<>();
+    commitMetadata.getExtraMetadata().forEach((k, v) -> {
+      if (k.startsWith(RecordIndexMetricNames.COMMIT_METADATA_PREFIX)) {
+        rli.put(k, v);
+      }
+    });
+    return rli;
+  }
+
+  private static String tagKey(String metric) {
+    return RecordIndexMetricNames.COMMIT_METADATA_PREFIX
+        + RecordIndexMetricNames.key(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric);
+  }
+
+  @ParameterizedTest
+  @ValueSource(booleans = {false, true})
+  public void testRliCountersReachCommitMetadataOnStreamerPath(boolean partitioned) throws Exception {
+    String label = partitioned ? "partitioned" : "global";
+    String tableBasePath = basePath + "/test_rli_metrics_streamer_" + label;
+
+    // Sync 1 -- build the table and the record index.
+    HoodieDeltaStreamer.Config insertCfg =
+        TestHoodieDeltaStreamer.TestHelpers.makeConfig(tableBasePath, WriteOperationType.INSERT);
+    enableRecordIndex(insertCfg, partitioned);
+    new HoodieDeltaStreamer(insertCfg, jsc).sync();
+
+    // Sync 2 -- upsert, which tags incoming keys against the record index.
+    HoodieDeltaStreamer.Config upsertCfg =
+        TestHoodieDeltaStreamer.TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT);
+    enableRecordIndex(upsertCfg, partitioned);
+    new HoodieDeltaStreamer(upsertCfg, jsc).sync();
+
+    Map<String, String> counters = rliCountersOnLatestCommit(tableBasePath);
+
+    System.out.println("\n===== DeltaStreamer (" + label + " RLI) -- RLI counters on the commit =====");
+    if (counters.isEmpty()) {
+      System.out.println("  (none found)");
+    } else {
+      counters.entrySet().stream()
+          .sorted(Map.Entry.comparingByKey())
+          .forEach(e -> System.out.println(String.format("  %-52s %s", e.getKey(), e.getValue())));
+    }
+    System.out.println("==========================================================\n");
+
+    assertFalse(counters.isEmpty(),
+        "the commit-boundary drain must fire on the DeltaStreamer path; hudi-utilities never calls "
+            + "Metrics.shutdownAllMetrics, so nothing else would publish these");
+
+    String lookedUp = tagKey(RecordIndexMetricNames.KEY_COUNT);
+    assertTrue(counters.containsKey(lookedUp),
+        "tag-location traffic must be attributed on the streamer path too; got " + counters.keySet());
+
+    long records = Long.parseLong(counters.get(lookedUp));
+    long hits = Long.parseLong(counters.get(tagKey(RecordIndexMetricNames.KEY_HIT_COUNT)));
+    long misses = Long.parseLong(counters.get(tagKey(RecordIndexMetricNames.KEY_MISS_COUNT)));
+    assertTrue(records > 0, "the upsert sync looked up at least one key");
+    assertEquals(records, hits + misses, "hits + misses must account for every key looked up");

Review Comment:
   Fixed, took the suggestion. You are right that `records == hits + misses` holds by construction -- `misses` is derived as `records - hits` at the emission site -- so it could not have caught a doubled count. Now asserts 1000 / 500 / 500 exactly.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la

Review Comment:
   Fixed -- the sentence was truncated. Rewritten, and it now describes reading off the reporter rather than the commit, since the commit-metadata sink is gone.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la
+ */
+abstract class RliLookupMetricsTestBase extends RecordLevelIndexTestBase {
+
+  /** Overridden by the partitioned subclasses; both variants are separate closures on separate paths. */
+  protected def isPartitionedRli: Boolean = false
+
+  /**
+   * Table type under test. Tagging is an index-level concern and does not branch on table type, so MOR
+   * is expected to behave identically -- the MOR subclasses exist to prove that rather than assume it.
+   */
+  protected def tableTypeOpt: String = DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL
+
+  protected def indexLabel: String = {
+    val idx = if (isPartitionedRli) "partitioned RLI" else "global RLI"
+    val tt = if (tableTypeOpt == DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL) "MOR" else "COW"
+    s"$idx, $tt"
+  }
+
+  /**
+   * `commonOpts` turns the global record index on, so the metadata-partition flags and the index type
+   * have to be flipped together to select the partitioned variant.
+   */
+  protected def rliOpts: Map[String, String] = {
+    val withTableType = Map(DataSourceWriteOptions.TABLE_TYPE.key -> tableTypeOpt)
+    if (isPartitionedRli) {
+      commonOpts ++ withTableType ++ Map(
+        HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false",
+        HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true",
+        HoodieIndexConfig.INDEX_TYPE.key -> "RECORD_LEVEL_INDEX")
+    } else {
+      commonOpts ++ withTableType ++ Map(
+        HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true",
+        HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false",
+        HoodieIndexConfig.INDEX_TYPE.key -> "GLOBAL_RECORD_LEVEL_INDEX")
+    }
+  }
+
+  protected def counterKey(caller: String, metric: String): String =
+    RecordIndexMetricNames.COMMIT_METADATA_PREFIX + RecordIndexMetricNames.key(caller, metric)
+
+  protected def tagKey(metric: String): String =
+    counterKey(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric)
+
+  /**
+   * A caller that looked something up stamps its full counter set, zeros included, so an absent key means
+   * that caller contributed nothing at all. The default is defensive against exactly that case.
+   */
+  protected def counterOrZero(counters: Map[String, String], caller: String, metric: String): Long =
+    counters.getOrElse(counterKey(caller, metric), "0").toLong
+
+  /** The counters as an operator would read them: off the latest completed commit. */
+  protected def rliCountersFromLatestCommit(): Map[String, String] = {
+    metaClient.reloadActiveTimeline()
+    val lastInstant = metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants().lastInstant().get()
+    metaClient.getActiveTimeline.readCommitMetadata(lastInstant).getExtraMetadata.asScala.toMap
+      .filter { case (k, _) => k.startsWith(RecordIndexMetricNames.COMMIT_METADATA_PREFIX) }
+  }
+
+  /** Leftover counters from a previous write would otherwise be folded into the next commit. */
+  protected def clearRliRegistry(): Unit = {
+    Registry.REGISTRY_MAP.asScala.foreach {
+      case (key, registry) => if (key.contains(RecordIndexMetricNames.REGISTRY_NAME)) registry.clear()
+    }
+  }
+
+  /** Asserts the core invariant and returns the looked-up count. */
+  protected def assertSumInvariant(counters: Map[String, String], caller: String): Long = {
+    val lookedUp = counterOrZero(counters, caller, RecordIndexMetricNames.KEY_COUNT)
+    val hits = counterOrZero(counters, caller, RecordIndexMetricNames.KEY_HIT_COUNT)
+    val misses = counterOrZero(counters, caller, RecordIndexMetricNames.KEY_MISS_COUNT)
+    org.junit.jupiter.api.Assertions.assertEquals(lookedUp, hits + misses,
+      s"hits + misses must account for every key looked up by '$caller'")
+    // A caller that looked something up must also report the time it took, or the timing metric is
+    // silently absent on paths nobody checked. Zero is allowed: a shard read can round below a millisecond.
+    if (lookedUp > 0) {
+      org.junit.jupiter.api.Assertions.assertTrue(
+        counters.contains(counterKey(caller, RecordIndexMetricNames.LOOKUP_TIME)),
+        s"'$caller' looked up $lookedUp keys but reported no ${RecordIndexMetricNames.LOOKUP_TIME}; " +
+          s"counters were ${counters.keys.toSeq.sorted.mkString(", ")}")
+      org.junit.jupiter.api.Assertions.assertTrue(
+        counterOrZero(counters, caller, RecordIndexMetricNames.LOOKUP_TIME) >= 0L,

Review Comment:
   Took the first half: the `>= 0L` assertion is dropped, since an unsigned counter cannot fail it and the `contains` above is the real check.
   
   Keeping the `report(...)` helpers for now. They are the only way the actual counter values reach the CI log, and on a metrics feature that is what makes a failure diagnosable rather than just a boolean. Happy to drop them if you would rather keep the log clean.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/PartitionedRecordIndexFileGroupLookupFunction.java:
##########
@@ -44,36 +47,57 @@ public class PartitionedRecordIndexFileGroupLookupFunction
     implements PairFlatMapFunction<Iterator<Pair<String, String>>, String, HoodieRecordGlobalLocation> {
 
   private final HoodieTableMetadata metadataTable;
+  // Empty when no counters should be collected; see RecordIndexLookupMetrics#resolveBundle.
+  private final Map<String, Registry> metricsBundle;
+  private final String caller;
 
+  /** Uninstrumented, for the query-side read path. */
   public PartitionedRecordIndexFileGroupLookupFunction(HoodieTableMetadata metadataTable) {
+    this(metadataTable, Collections.emptyMap(), null);
+  }
+
+  public PartitionedRecordIndexFileGroupLookupFunction(HoodieTableMetadata metadataTable,
+                                                       Map<String, Registry> metricsBundle, String caller) {
     this.metadataTable = metadataTable;
+    this.metricsBundle = metricsBundle;
+    this.caller = caller;
   }
 
   @Override
   public Iterator<Tuple2<String, HoodieRecordGlobalLocation>> call(Iterator<Pair<String, String>> partitionPathRecordKeyIterator) {
-    String partitionName = null;
-    List<String> keysToLookup = new ArrayList<>();
-    while (partitionPathRecordKeyIterator.hasNext()) {
-      Pair<String, String> partitionPathRecordKey = partitionPathRecordKeyIterator.next();
-      keysToLookup.add(partitionPathRecordKey.getRight());
-      if (partitionName == null) {
-        partitionName = partitionPathRecordKey.getLeft();
+    // Bound for the whole task so a metric raised deeper in the lookup resolves here too.

Review Comment:
   Fixed, took the suggestion -- the binding is released in the `finally` before Spark consumes the iterator, so it covers the shard read, not the whole task.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5420085787

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * 221b2b5bfec5eb6548462bd91fe85b3502f52637 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556) 
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 308c956e2a5895f99806e8d2d242f318cf670c42 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3862767500


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {

Review Comment:
   The DataSource half holds only because `DefaultSource:194` calls `Metrics.shutdownAllMetrics()` after every write. `UpdateHoodieTableCommand:71`, `MergeIntoHoodieTableCommand:509` and `StreamSync` call `HoodieSparkSqlWriter.write` / the client directly and never shut metrics down between commits, so on those paths an abandoned attempt's counters still land on the next successful commit, and the flipped test runs the DataSource path only. Could we clear the group's registry at `startCommit` (your `dd4a48c`), which makes the guarantee path-independent, and run the failed-commit case once on the SQL class?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858495125


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {

Review Comment:
   this will be removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5451879917

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     }, {
       "hash" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644",
       "triggerID" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bc8688a911937a4a0fef69304f182697141245d9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16651",
       "triggerID" : "bc8688a911937a4a0fef69304f182697141245d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "29636dd2c30bd4c18507736bec602d938274ceb9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "29636dd2c30bd4c18507736bec602d938274ceb9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * bc8688a911937a4a0fef69304f182697141245d9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16651) 
   * 29636dd2c30bd4c18507736bec602d938274ceb9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5419600057

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e62a131d0a4821a474fc78f037df1c82fcc1c559 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555) 
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5247401127

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0fd521723cb8a9ce999369619fe454e5ca47b4c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5247367567

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0fd521723cb8a9ce999369619fe454e5ca47b4c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5248100752

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0fd521723cb8a9ce999369619fe454e5ca47b4c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5249199618

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c4fa5fad9b2a0ec0866cd13778d7288c22275494 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5363584415

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3d9cb9c3c5e0631abdad072dd0a6706ac42d4521 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428) 
   * 30dbc00d04a21cd671b1e02b527f3b340f828008 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3877789261


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -146,8 +147,16 @@ public boolean commit(String instantTime, JavaRDD<WriteStatus> rawWriteStatuses,
       // when streaming writes are enabled, writeStatuses is a mix of data table write status and mdt write status
       List<HoodieWriteStat> dataTableHoodieWriteStats = slimWriteStatsList.stream().filter(entry -> !entry.isMetadataTable()).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
       List<HoodieWriteStat> partialMetadataTableWriteStats = slimWriteStatsList.stream().filter(entry -> entry.isMetadataTable).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
-      return commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds, extraPreCommitFunc,
-          false, Option.of(table));
+      boolean committed = commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds,
+          extraPreCommitFunc, false, Option.of(table));
+      // Publish only if this write tagged. requiresTagging is the same predicate BaseWriteHelper uses
+      // to decide whether to look up at all, so publishing mirrors collecting: an INSERT or a
+      // BULK_INSERT never collects, and must not publish counters an earlier write left behind.
+      if (committed && getOperationType() != null

Review Comment:
   and it seems that `commitStats(` will never return false, it either return true or throws.
   
   also can we move the rli lookup metrics publish into the `postCommit` method? since that is where the metrics publish should be located.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5364568860

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2c37a62e311d57eb3636dc5f28af1739788c98f9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432) 
   * a061dbc0124d8ebc62defdf41d2d428842a6b16c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5261153045

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5d3fa9e8ab489adcd8b266911cfaee7672ab8846 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5248657593

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f0fd521723cb8a9ce999369619fe454e5ca47b4c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207) 
   * c4fa5fad9b2a0ec0866cd13778d7288c22275494 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5274871764

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f2ace685b8450e4466990371e11deba541aa992 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234) 
   * bb09a2d96ee77c288bc295a9195cfca97a81e3bd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259) 
   * 369fbd591652f70be61c8c5fa19526955d41ba38 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5364191017

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 30dbc00d04a21cd671b1e02b527f3b340f828008 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431) 
   * 2c37a62e311d57eb3636dc5f28af1739788c98f9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5363894902

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3d9cb9c3c5e0631abdad072dd0a6706ac42d4521 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428) 
   * 30dbc00d04a21cd671b1e02b527f3b340f828008 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431) 
   * 2c37a62e311d57eb3636dc5f28af1739788c98f9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): make record index lookups observable via executor metrics [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5362246857

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 70968d47931e1e491260eccd0eadd85e12a83202 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863057489


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la
+ */
+abstract class RliLookupMetricsTestBase extends RecordLevelIndexTestBase {

Review Comment:
   Two of the seven are in. Of the rest, the MOR twins are the cheapest: nothing under `RecordIndexLookupMetrics`, `ExecutorMetrics`, `DistributedRegistry` or the two lookup functions branches on table type, and both table types reach the drain through the same `commitStats`, so they re-run 22 writes over identical lines. The multi-table class now separates gauges by `METRICS_REPORTER_PREFIX`, which is the table name, so it still cannot exercise the digest (`countersAreScopedByBasePathNotOnlyByTableName` does). Could those two go now, leaving the partitioned twins until the lifetime settles?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5433561407

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 0a9d4b412caeb2ef3dc19c1b16287265bd106fe7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568) 
   * eadd7bbf6072d5c29c93f1650ae25f2b5f4de998 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3870639302


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  /**
+   * The registry a lookup task collects into, or null when nothing should be collected. Captured in the
+   * lookup closure and passed to {@link #recordShardLookup}, so delivery is by closure capture rather
+   * than by name.
+   *
+   * <p>Requires the reporter to be on as well: with {@code hoodie.metrics.on} off there is nowhere to
+   * publish, and collecting would register an accumulator and scan every shard for nothing.
+   */
+  public static Registry resolveRegistry(HoodieEngineContext context, HoodieWriteConfig config) {
+    if (!config.isMetricsOn() || !ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.isEnabled(config)) {
+      return null;
+    }
+    // TBL_NAME has no default and Builder.validate() only requires BASE_PATH, so a config built without
+    // forTable() reaches here with a null name. getMetricRegistry dereferences it immediately.
+    String tableName = config.getTableName();
+    if (tableName == null || tableName.isEmpty()) {
+      return null;
+    }
+    Registry registry = context.getMetricRegistry(tableName,
+        ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.scopedName(config.getBasePath()));
+    // Only the accumulator-backed registry aggregates back to the driver. A LocalRegistry here would
+    // collect on the executor and be dropped on the floor, so report nothing instead.
+    if (!(registry instanceof DistributedRegistry)) {
+      return null;
+    }
+    // Runs on the driver before the closure ships, so anything held now predates this write. Publishing
+    // releases what it reports, which leaves a non-empty registry only after an attempt that never
+    // committed. Paths that tear metrics down between writes lose it anyway; Spark SQL DML and StreamSync
+    // do not, and this commit must not report work it did not do.
+    registry.clear();

Review Comment:
   [P2] Do not clear a table-shared accumulator at lookup construction
   
   The registry key is shared by every write client for this table, so this clear cannot establish a per-commit boundary when writers overlap. For example, writer A can finish its lookup, writer B can clear those counts here, and then either commit can publish zero, the other writer counts, or a mixture. That makes the advertised per-commit counters incorrect under supported multiwriter operation. Could the accumulator be scoped by the in-flight write or instant, or could collection be explicitly disabled for multiwriter configurations until that scoping exists?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5447641239

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     }, {
       "hash" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644",
       "triggerID" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 695f74f6364193e7869f83b7eab88e9b637b555c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858225376


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -255,6 +256,12 @@ public boolean commitStats(String instantTime, TableWriteStats tableWriteStats,
       return true;
     }
     extraMetadata = updateExtraMetadata(extraMetadata);
+    // The commit is the boundary for the record index lookup counters. Snapshot them into this commit's
+    // metadata now, but only release them from the registry once the commit has actually landed --
+    // hooked here rather than in updateExtraMetadata because that is shared with table-service
+    // scheduling, which would otherwise consume the counters into a compaction or clustering plan.
+    ExecutorMetrics.DrainedCounters executorCounters =
+        ExecutorMetrics.snapshotIntoCommitMetadata(extraMetadata.get(), config);

Review Comment:
   I am debating on if we should even be tracking these new metrics within the commit metadata, or if we should fully delegate the responsibility to metrics reporter in order to reduce the complexity here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859018665


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRliLookupMetricsReporting.scala:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.config.metrics.HoodieMetricsConfig
+import org.apache.hudi.metrics.{ExecutorMetricRegistry, RecordIndexMetricNames}
+
+import org.apache.spark.sql.SaveMode
+import org.junit.jupiter.api.{Tag, Test}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue}
+
+import java.io.{ByteArrayOutputStream, PrintStream}
+
+/** The counters must reach a live metrics reporter, not only commit metadata. */
+@Tag("functional")
+class TestRliLookupMetricsReporting extends RliLookupMetricsTestBase {
+
+  private def metricsOpts: Map[String, String] = rliOpts ++ Map(
+    HoodieMetricsConfig.TURN_METRICS_ON.key -> "true",
+    HoodieMetricsConfig.METRICS_REPORTER_TYPE_VALUE.key -> "CONSOLE")
+
+  /** Captures stdout for the duration of the write. */
+  private def captureStdout(body: => Unit): String = {
+    val buffer = new ByteArrayOutputStream()
+    val original = System.out
+    try {
+      System.setOut(new PrintStream(buffer, true, "UTF-8"))
+      body
+    } finally {
+      System.setOut(original)
+    }
+    buffer.toString("UTF-8")
+  }
+
+  @Test
+  def testCountersReachBothCommitMetadataAndTheReporter(): Unit = {
+    val numUpdates = 12
+
+    doWriteAndValidateDataAndRecordIndex(metricsOpts, INSERT_OPERATION_OPT_VAL, SaveMode.Overwrite,
+      validate = false, numInserts = 50)
+    clearRliRegistry()
+
+    val stdout = captureStdout {
+      doWriteAndValidateDataAndRecordIndex(metricsOpts, UPSERT_OPERATION_OPT_VAL, SaveMode.Append,
+        validate = false, numUpdates = numUpdates)
+    }
+
+    // Sink 1 -- the timeline.
+    val counters = rliCountersFromLatestCommit()
+    report(s"Reporter test ($indexLabel) -- commit metadata", counters)
+    assertTrue(counters.nonEmpty, "the commit must still carry the counters when a reporter is configured")
+    assertEquals(numUpdates.toString, counters(tagKey(RecordIndexMetricNames.KEY_HIT_COUNT)))
+    assertEquals((numUpdates + 1).toLong, assertSumInvariant(counters, RecordIndexMetricNames.CALLER_TAG_LOCATION))
+
+    // Sink 2 -- the reporter. Gauge names are <prefix>.rli.lookup.<caller>.<metric>.
+    val gaugePrefix = s"${ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.metricAction}.${ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.metricQualifier}"
+    val reported = stdout.linesIterator.filter(_.contains(gaugePrefix)).toSeq
+
+    println(s"\n===== Reporter test ($indexLabel) -- ConsoleMetricsReporter output =====")
+    if (reported.isEmpty) println("  (no rli.lookup gauges printed)") else reported.foreach(l => println(s"  ${l.trim}"))
+    println("=======================================================================\n")
+
+    assertTrue(reported.nonEmpty,
+      s"ConsoleMetricsReporter must publish the '$gaugePrefix' gauges; the drain feeds the reporter and " +
+        "commit metadata from a single read, so finding them in the commit but not here means the " +
+        "reporter sink regressed")
+    Seq(RecordIndexMetricNames.KEY_HIT_COUNT, RecordIndexMetricNames.KEY_MISS_COUNT,
+      RecordIndexMetricNames.KEY_COUNT, RecordIndexMetricNames.SHARDS_READ).foreach { metric =>
+      val name = s"$gaugePrefix.${RecordIndexMetricNames.key(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric)}"
+      assertTrue(stdout.contains(name), s"reporter output must contain the gauge '$name'")

Review Comment:
   Resolved -- `TestRliLookupMetricsReporting` is deleted. Reporting is now the only sink, so every remaining functional test reads values back off a reporter and asserts them, which is the stronger check you were asking for.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la
+ */
+abstract class RliLookupMetricsTestBase extends RecordLevelIndexTestBase {
+
+  /** Overridden by the partitioned subclasses; both variants are separate closures on separate paths. */
+  protected def isPartitionedRli: Boolean = false
+
+  /**
+   * Table type under test. Tagging is an index-level concern and does not branch on table type, so MOR
+   * is expected to behave identically -- the MOR subclasses exist to prove that rather than assume it.
+   */
+  protected def tableTypeOpt: String = DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL
+
+  protected def indexLabel: String = {
+    val idx = if (isPartitionedRli) "partitioned RLI" else "global RLI"
+    val tt = if (tableTypeOpt == DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL) "MOR" else "COW"
+    s"$idx, $tt"
+  }
+
+  /**
+   * `commonOpts` turns the global record index on, so the metadata-partition flags and the index type
+   * have to be flipped together to select the partitioned variant.
+   */
+  protected def rliOpts: Map[String, String] = {
+    val withTableType = Map(DataSourceWriteOptions.TABLE_TYPE.key -> tableTypeOpt)
+    if (isPartitionedRli) {
+      commonOpts ++ withTableType ++ Map(
+        HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false",
+        HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true",
+        HoodieIndexConfig.INDEX_TYPE.key -> "RECORD_LEVEL_INDEX")
+    } else {
+      commonOpts ++ withTableType ++ Map(
+        HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true",
+        HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false",
+        HoodieIndexConfig.INDEX_TYPE.key -> "GLOBAL_RECORD_LEVEL_INDEX")
+    }
+  }
+
+  protected def counterKey(caller: String, metric: String): String =
+    RecordIndexMetricNames.COMMIT_METADATA_PREFIX + RecordIndexMetricNames.key(caller, metric)
+
+  protected def tagKey(metric: String): String =
+    counterKey(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric)
+
+  /**
+   * A caller that looked something up stamps its full counter set, zeros included, so an absent key means
+   * that caller contributed nothing at all. The default is defensive against exactly that case.
+   */
+  protected def counterOrZero(counters: Map[String, String], caller: String, metric: String): Long =
+    counters.getOrElse(counterKey(caller, metric), "0").toLong
+
+  /** The counters as an operator would read them: off the latest completed commit. */
+  protected def rliCountersFromLatestCommit(): Map[String, String] = {
+    metaClient.reloadActiveTimeline()
+    val lastInstant = metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants().lastInstant().get()
+    metaClient.getActiveTimeline.readCommitMetadata(lastInstant).getExtraMetadata.asScala.toMap
+      .filter { case (k, _) => k.startsWith(RecordIndexMetricNames.COMMIT_METADATA_PREFIX) }
+  }
+
+  /** Leftover counters from a previous write would otherwise be folded into the next commit. */
+  protected def clearRliRegistry(): Unit = {

Review Comment:
   Fixed -- `clearRliRegistry()` and all its call sites are deleted. You were right that `@TempDir` being per method makes the base-path digest unique per test, so it could never have been doing anything.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = context.getMetricRegistry(config.getTableName(),
+          metricRegistry.scopedName(config.getBasePath()));
+      // Only the accumulator-backed registry aggregates back to the driver, so anything else is left out
+      // rather than bound: a bound LocalRegistry would collect on the executor and be dropped on the floor,
+      // whereas leaving it out makes the lookup resolve to a no-op that reports nothing.
+      if (registry instanceof DistributedRegistry) {
+        bundle.put(metricRegistry.registryName(), registry);
+      }
+    }
+    return bundle.isEmpty() ? Collections.emptyMap() : bundle;
+  }
+
+  /**
+   * Records one shard's lookup outcome. Counts records rather than distinct keys, so
+   * {@code hits + misses == records_looked_up} holds when a batch repeats a key. Membership is tested
+   * against the found set, bounded by the hit count, not the asked-about set, bounded by shard size.
+   *
+   * @param keysLookedUp every record key routed to this shard
+   * @param foundKeys    the subset present in the index
+   * @param elapsedMs    wall-clock spent reading this shard
+   */
+  public static void recordShardLookup(String caller, Collection<String> keysLookedUp,
+                                       Collection<String> foundKeys, long elapsedMs) {
+    if (keysLookedUp.isEmpty()) {
+      return;
+    }
+    Registry registry = Registry.getRegistry(RecordIndexMetricNames.REGISTRY_NAME);

Review Comment:
   Deferring, but this is the one I most want to come back to. You are right that nothing shipped emits without the bundle in hand, and that passing the `Registry` in directly would let `ExecutorMetricsContext`, `NoOpRegistry`, `TestRegistryExecutorLookup`, the `HoodieSparkTable` binding and the `Registry.getRegistry` contract change all go, both hudi-io additions included.
   
   Not taking it here for one reason: name resolution is the part the requester specifically asked for, so that code below the write API can take part without a signature change. Collapsing it to a passed-in `Registry` is the right call if that requirement softens, and it is a much larger delete than anything else on this PR, so it deserves its own change rather than being folded into a review pass. Filed as a follow-up.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {
+    DISTRIBUTED_REGISTRY_MAP.remove(tableName.isEmpty() ? registryName : tableName + "." + registryName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {

Review Comment:
   Deferring to the follow-up, agreed on the diagnosis. The process-wide map is what forces the digest, the stale-context branch, the replacement branch and the remover, and it is where the unbounded growth lives -- I reproduced 40+ never-evicted entries in one functional run. #19063 already lists instance-scoped registry lifetime as the intended next step and your pointers (`StreamSync:1104`, `client.getEngineContext` in scope at `HoodieSparkSqlWriter:553`) look right. It changes the lifetime model for an existing shared component, so it wants its own PR and its own review rather than riding along here.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/DistributedRegistry.java:
##########
@@ -48,9 +51,17 @@ public String getName() {
   public void register(JavaSparkContext jsc) {
     if (!isRegistered()) {
       jsc.sc().register(this);
+      // Only when this call actually registers: stamping unconditionally would re-brand an accumulator
+      // bound to a dead context and mask the staleness this field exists to detect.
+      this.registeredAppId = jsc.sc().applicationId();
     }
   }
 
+  /** False when bound to a different (typically stopped) context, meaning it must be recreated. */
+  public boolean isRegisteredWith(JavaSparkContext jsc) {

Review Comment:
   Agreed that the `DistributedRegistry` / `getMetricRegistry` hardening is separable, and that it fixes `hoodie.metrics.executor.enable` for the existing `HoodieWrapperFileSystem` user independent of anything RLI. It is about 830 lines across 11 files, a third of this PR.
   
   Holding off only because splitting now means restacking a PR that is close to landing, and the counters do not work without it. Adding the #19063 reference to the description in the meantime so the relationship is not lost.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5419639217

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e62a131d0a4821a474fc78f037df1c82fcc1c559 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555) 
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * 221b2b5bfec5eb6548462bd91fe85b3502f52637 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859109998


##########
hudi-common/src/main/java/org/apache/hudi/common/config/metrics/HoodieMetricsConfig.java:
##########


Review Comment:
   🤖 **Line 117:** This doc still describes the old behavior: the counters were written into the commit extra metadata, but this PR publishes them only to the metrics reporter. The warning about permanent timeline/archive footprint no longer applies. Could you reword it for reporter-only publishing? Also, since master metrics default off but this defaults to true, the executor accumulator still runs (reporting nothing) on every RLI table by default, so it may be worth a line on why one would still disable it.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3880058990


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The record index lookup counters, end to end: what they are called, where they are collected, and how
+ * they reach the reporter once a commit lands.
+ *
+ * <p>The numbers are only knowable on executors, because a lookup returns its hits and a miss produces
+ * no output row at all. So the driver resolves an accumulator-backed {@link DistributedRegistry}, the
+ * lookup closure captures it, tasks add into their own copy, and Spark merges those copies home.
+ */
+public class RecordIndexLookupMetrics {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RecordIndexLookupMetrics.class);
+
+  /** The registry name, as reported. Keying is done by {@link #registryKey}. */
+  public static final String REGISTRY_NAME = "HoodieRecordIndexLookup";
+
+  /** Reporter naming, as passed to {@code HoodieMetrics.getMetricsName}. */
+  public static final String METRIC_ACTION = "rli";
+  public static final String METRIC_QUALIFIER = "lookup";
+
+  /** Counts records, not distinct keys: a batch repeating a key contributes once per record, which is
+   * what keeps {@code hits + misses == key_count} exact. */
+  public static final String KEY_COUNT = "lookup_record_index_key_count";
+  public static final String KEY_HIT_COUNT = "lookup_record_index_key_hit_count";
+  public static final String KEY_MISS_COUNT = "lookup_record_index_key_miss_count";
+  public static final String SHARDS_READ = "lookup_record_index_shards_read";
+  /**
+   * Wall-clock spent in the shard read, summed across shards rather than averaged because shards are read
+   * in parallel: the value is per-commit read effort, and dividing by {@link #SHARDS_READ} gives a mean.
+   *
+   * <p>Distinct from {@code index.lookup.duration} published by {@code HoodieMetrics.updateIndexMetrics},
+   * which is driver wall-clock for the whole {@code tagLocation} including scheduling. Comparing the two
+   * shows how much of a lookup was actually spent reading the index.
+   */
+  public static final String LOOKUP_TIME = "lookup_record_index_time";
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  /**
+   * The registry a lookup task collects into, or null when nothing should be collected. Captured in the
+   * lookup closure and passed to {@link #recordShardLookup}, so delivery is by closure capture rather
+   * than by name.
+   *
+   * <p>Requires the reporter to be on as well: with {@code hoodie.metrics.on} off there is nowhere to
+   * publish, and collecting would register an accumulator and scan every shard for nothing.
+   */
+  public static Option<Registry> resolveRegistry(HoodieEngineContext context, HoodieWriteConfig config) {
+    if (!config.isMetricsOn() || !config.isRecordIndexLookupMetricsEnabled()
+        || !(context instanceof HoodieSparkEngineContext)) {
+      return Option.empty();
+    }
+    DistributedRegistry registry = ((HoodieSparkEngineContext) context)
+        .getOrCreateOwnedRegistry(registryKey(config.getBasePath()), REGISTRY_NAME);
+    // Anything held predates this write: the drain removes the entry as it publishes, so a surviving
+    // one belongs to an attempt that never committed.
+    registry.clear();
+    return Option.of(registry);
+  }
+
+  /**
+   * Records one shard's lookup outcome. Counts records rather than distinct keys, so
+   * {@code hits + misses == records_looked_up} holds when a batch repeats a key. Membership is tested
+   * against the found set, bounded by the hit count, not the asked-about set, bounded by shard size.
+   *
+   * <p>Callers must skip the call entirely when the registry is null. Building the found set is
+   * O(hits), and an argument is evaluated before this method can take a null fast path, so guarding
+   * here would not keep that cost off the disabled path.
+   *
+   * @param registry     where to collect, never null
+   * @param keysLookedUp every record key routed to this shard
+   * @param foundKeys    the subset present in the index
+   * @param elapsedMs    wall-clock spent reading this shard
+   */
+  public static void recordShardLookup(Registry registry, Collection<String> keysLookedUp,
+                                       Collection<String> foundKeys, long elapsedMs) {
+    if (keysLookedUp.isEmpty()) {
+      return;
+    }
+    Set<String> found = foundKeys instanceof Set ? (Set<String>) foundKeys : new HashSet<>(foundKeys);
+    long records = keysLookedUp.size();
+    long hits = found.isEmpty() ? 0L : keysLookedUp.stream().filter(found::contains).count();
+    registry.add(KEY_COUNT, records);
+    registry.add(KEY_HIT_COUNT, hits);
+    registry.add(KEY_MISS_COUNT, records - hits);
+    registry.increment(SHARDS_READ);
+    registry.add(LOOKUP_TIME, elapsedMs);
+  }
+
+  /**
+   * Reports the counters and releases them, so the next commit reports only its own work. Called once
+   * the commit has landed, so a commit that never lands publishes nothing.
+   *
+   * <p>Release subtracts what was reported rather than clearing, so a straggler task whose update lands
+   * mid-publish carries into the next commit instead of being dropped. An all-zero registry is skipped.
+   */
+  public static void publishAndRelease(HoodieEngineContext context, HoodieWriteConfig config,
+                                       HoodieMetrics hoodieMetrics) {
+    try {
+      publish(context, config, hoodieMetrics);
+    } catch (Exception e) {
+      // This runs after the commit has landed. Reporting is not worth failing a completed write over.
+      LOG.warn("Failed to publish record index lookup metrics; the commit is unaffected.", e);
+    }
+  }
+
+  private static void publish(HoodieEngineContext context, HoodieWriteConfig config,
+                              HoodieMetrics hoodieMetrics) {
+    if (!config.isRecordIndexLookupMetricsEnabled()) {
+      // Callers gate before reaching here, so this is a wiring mistake rather than normal flow.
+      LOG.warn("Record index lookup metrics drain reached with the feature disabled; nothing published.");
+      return;
+    }
+    if (!(context instanceof HoodieSparkEngineContext)) {
+      return;
+    }
+    // Removing is what makes attribution work. An entry exists only because a lookup on this write
+    // created it, so its absence means this write looked nothing up and has nothing of its own to
+    // report -- which is the case an operation type cannot tell apart, since an insert that drops
+    // duplicates tags through SparkRDDReadClient without requiring tagging.
+    Option<DistributedRegistry> taken =
+        ((HoodieSparkEngineContext) context).removeOwnedRegistry(registryKey(config.getBasePath()));

Review Comment:
   Fixed in 29636dd. `handleInsertDuplicates` now passes the committing client's engine context and config into `DataSourceUtils.handleDuplicates`, dropping the throwaway `new HoodieSparkEngineContext(jssc)` that `resolveDuplicates` built, so the dedup lookup registry is owned by the same context `postCommit` drains. Added `TestRliLookupMetricsOnDataSource#testInsertDropDupsPublishesDedupLookupCounters` (inherited by the partitioned variant); it fails on the old wiring (no counters published) and passes now, reporting key_count/hits/misses = 55/40/15.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5452094476

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     }, {
       "hash" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644",
       "triggerID" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bc8688a911937a4a0fef69304f182697141245d9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16651",
       "triggerID" : "bc8688a911937a4a0fef69304f182697141245d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "29636dd2c30bd4c18507736bec602d938274ceb9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16669",
       "triggerID" : "29636dd2c30bd4c18507736bec602d938274ceb9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * bc8688a911937a4a0fef69304f182697141245d9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16651) 
   * 29636dd2c30bd4c18507736bec602d938274ceb9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16669) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5262991232

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5d3fa9e8ab489adcd8b266911cfaee7672ab8846 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227) 
   * 4f2ace685b8450e4466990371e11deba541aa992 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5274503111

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f2ace685b8450e4466990371e11deba541aa992 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234) 
   * bb09a2d96ee77c288bc295a9195cfca97a81e3bd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3877796191


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,232 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The record index lookup counters, end to end: what they are called, where they are collected, and how
+ * they reach the reporter once a commit lands.
+ *
+ * <p>The numbers are only knowable on executors, because a lookup returns its hits and a miss produces
+ * no output row at all. So the driver resolves an accumulator-backed {@link DistributedRegistry}, the
+ * lookup closure captures it, tasks add into their own copy, and Spark merges those copies home.
+ */
+public class RecordIndexLookupMetrics {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RecordIndexLookupMetrics.class);
+
+  /** The registry name. Scoping for the driver-side key is applied by {@link #scopedName}. */
+  public static final String REGISTRY_NAME = "HoodieRecordIndexLookup";
+
+  /** Reporter naming, as passed to {@code HoodieMetrics.getMetricsName}. */
+  public static final String METRIC_ACTION = "rli";
+  public static final String METRIC_QUALIFIER = "lookup";
+
+  /** Counts records, not distinct keys: a batch repeating a key contributes once per record, which is
+   * what keeps {@code hits + misses == key_count} exact. */
+  public static final String KEY_COUNT = "lookup_record_index_key_count";
+  public static final String KEY_HIT_COUNT = "lookup_record_index_key_hit_count";
+  public static final String KEY_MISS_COUNT = "lookup_record_index_key_miss_count";
+  public static final String SHARDS_READ = "lookup_record_index_shards_read";
+  /**
+   * Wall-clock spent in the shard read, summed across shards rather than averaged because shards are read
+   * in parallel: the value is per-commit read effort, and dividing by {@link #SHARDS_READ} gives a mean.
+   *
+   * <p>Distinct from {@code index.lookup.duration} published by {@code HoodieMetrics.updateIndexMetrics},
+   * which is driver wall-clock for the whole {@code tagLocation} including scheduling. Comparing the two
+   * shows how much of a lookup was actually spent reading the index.
+   */
+  public static final String LOOKUP_TIME = "lookup_record_index_time";
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  /**
+   * The registry a lookup task collects into, or null when nothing should be collected. Captured in the
+   * lookup closure and passed to {@link #recordShardLookup}, so delivery is by closure capture rather
+   * than by name.
+   *
+   * <p>Requires the reporter to be on as well: with {@code hoodie.metrics.on} off there is nowhere to
+   * publish, and collecting would register an accumulator and scan every shard for nothing.
+   */
+  public static Registry resolveRegistry(HoodieEngineContext context, HoodieWriteConfig config) {

Review Comment:
   make it return Option instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5364969196

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a061dbc0124d8ebc62defdf41d2d428842a6b16c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435) 
   * c3df340e3609cb1ad7f5f8abce5d207625887fa7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5363226351

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 70968d47931e1e491260eccd0eadd85e12a83202 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426) 
   * 3d9cb9c3c5e0631abdad072dd0a6706ac42d4521 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428) 
   * 30dbc00d04a21cd671b1e02b527f3b340f828008 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859286572


##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestRliMetricsOnStreamerPath.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.deltastreamer;
+
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The RLI lookup counters must reach commit metadata on the DeltaStreamer path, not only on the Spark DataSource path.
+ */
+@Tag("functional")
+public class TestRliMetricsOnStreamerPath extends HoodieDeltaStreamerTestBase {
+
+  /** Selects the global or partitioned record index. */
+  private static void enableRecordIndex(HoodieDeltaStreamer.Config cfg, boolean partitioned) {
+    cfg.configs.add(HoodieMetadataConfig.ENABLE.key() + "=true");
+    cfg.configs.add(HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key() + "=" + !partitioned);
+    cfg.configs.add(HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key() + "=" + partitioned);
+    cfg.configs.add(HoodieIndexConfig.INDEX_TYPE.key() + "="
+        + (partitioned ? "RECORD_LEVEL_INDEX" : "GLOBAL_RECORD_LEVEL_INDEX"));
+  }
+
+  private static Map<String, String> rliCountersOnLatestCommit(String tableBasePath) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HoodieTestUtils.getDefaultStorageConf())
+        .setBasePath(tableBasePath)
+        .build();
+    metaClient.reloadActiveTimeline();
+    HoodieInstant lastInstant = metaClient.getActiveTimeline()
+        .getCommitsTimeline().filterCompletedInstants().lastInstant().get();
+    HoodieCommitMetadata commitMetadata = metaClient.getActiveTimeline().readCommitMetadata(lastInstant);
+    Map<String, String> rli = new HashMap<>();
+    commitMetadata.getExtraMetadata().forEach((k, v) -> {
+      if (k.startsWith(RecordIndexMetricNames.COMMIT_METADATA_PREFIX)) {
+        rli.put(k, v);
+      }
+    });
+    return rli;
+  }
+
+  private static String tagKey(String metric) {
+    return RecordIndexMetricNames.COMMIT_METADATA_PREFIX
+        + RecordIndexMetricNames.key(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric);
+  }
+
+  @ParameterizedTest
+  @ValueSource(booleans = {false, true})
+  public void testRliCountersReachCommitMetadataOnStreamerPath(boolean partitioned) throws Exception {
+    String label = partitioned ? "partitioned" : "global";
+    String tableBasePath = basePath + "/test_rli_metrics_streamer_" + label;
+
+    // Sync 1 -- build the table and the record index.
+    HoodieDeltaStreamer.Config insertCfg =
+        TestHoodieDeltaStreamer.TestHelpers.makeConfig(tableBasePath, WriteOperationType.INSERT);
+    enableRecordIndex(insertCfg, partitioned);
+    new HoodieDeltaStreamer(insertCfg, jsc).sync();
+
+    // Sync 2 -- upsert, which tags incoming keys against the record index.
+    HoodieDeltaStreamer.Config upsertCfg =
+        TestHoodieDeltaStreamer.TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT);
+    enableRecordIndex(upsertCfg, partitioned);
+    new HoodieDeltaStreamer(upsertCfg, jsc).sync();
+
+    Map<String, String> counters = rliCountersOnLatestCommit(tableBasePath);
+
+    System.out.println("\n===== DeltaStreamer (" + label + " RLI) -- RLI counters on the commit =====");
+    if (counters.isEmpty()) {
+      System.out.println("  (none found)");
+    } else {
+      counters.entrySet().stream()
+          .sorted(Map.Entry.comparingByKey())
+          .forEach(e -> System.out.println(String.format("  %-52s %s", e.getKey(), e.getValue())));
+    }
+    System.out.println("==========================================================\n");
+
+    assertFalse(counters.isEmpty(),
+        "the commit-boundary drain must fire on the DeltaStreamer path; hudi-utilities never calls "
+            + "Metrics.shutdownAllMetrics, so nothing else would publish these");
+
+    String lookedUp = tagKey(RecordIndexMetricNames.KEY_COUNT);
+    assertTrue(counters.containsKey(lookedUp),
+        "tag-location traffic must be attributed on the streamer path too; got " + counters.keySet());
+
+    long records = Long.parseLong(counters.get(lookedUp));
+    long hits = Long.parseLong(counters.get(tagKey(RecordIndexMetricNames.KEY_HIT_COUNT)));
+    long misses = Long.parseLong(counters.get(tagKey(RecordIndexMetricNames.KEY_MISS_COUNT)));
+    assertTrue(records > 0, "the upsert sync looked up at least one key");
+    assertEquals(records, hits + misses, "hits + misses must account for every key looked up");

Review Comment:
   Now done properly, with the counts pinned as you asked: 1000 / 500 / 500.
   
   The reason I backed off earlier was wrong, and the test was worse than I thought. It failed outright on this path -- zero counters -- and I had never actually run it, because every attempt died before reaching the assertions. The cause was in the test, not the write path: it rebuilt a `HoodieMetrics` from the base path to read gauges, but `Metrics` is keyed by the base path on the *metrics* config, which that config did not carry, so `getInstance` minted a fresh empty registry instead of finding the sync's. Worse, `getInstance` creates on miss, so probing with `isInitialized` reported success on the instance the probe had itself just created.
   
   It now reads through a capturing reporter, independent of how `Metrics` is keyed, and matches gauges on the group's action and qualifier rather than a reconstructed prefix -- the prefix is the real table name, which the test cannot know.
   
   Verified locally: 2/2 green, counters 1000 / 500 / 500 on both the global and partitioned arms. `shards_read` is left as a lower bound since it follows the index file-group layout (10 global, 3 partitioned on this workload).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858916647


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {

Review Comment:
   im not sure if this is really needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863013178


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/DistributedRegistry.java:
##########
@@ -48,9 +51,17 @@ public String getName() {
   public void register(JavaSparkContext jsc) {
     if (!isRegistered()) {
       jsc.sc().register(this);
+      // Only when this call actually registers: stamping unconditionally would re-brand an accumulator
+      // bound to a dead context and mask the staleness this field exists to detect.
+      this.registeredAppId = jsc.sc().applicationId();
     }
   }
 
+  /** False when bound to a different (typically stopped) context, meaning it must be recreated. */
+  public boolean isRegisteredWith(JavaSparkContext jsc) {

Review Comment:
   The branch is already stacked for this: `ee961488` is the hardening on its own (`DistributedRegistry`, `getMetricRegistry`, `Registry` / `ExecutorMetricsContext` / `NoOpRegistry` and their four test classes, 811 lines, nothing RLI-specific), with `8c4d0c96` and `8a71f719` on top. It fixes `hoodie.metrics.executor.enable` for `HoodieWrapperFileSystem` on master regardless of where the counter design (name resolution, instance scoping) ends up, so could `ee961488` plus the guard and remover hunks from `05a5243` go out first as #19063's PR 1, with this PR rebased on it? That is one `rebase -i` rather than a restack, and it lets the fix land even if the counters take another round.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863085631


##########
hudi-io/src/main/java/org/apache/hudi/common/metrics/Registry.java:
##########
@@ -190,6 +199,28 @@ static void setRegistries(Collection<Registry> registries) {
    */
   void set(String name, long value);
 
+  /**
+   * Subtract a set of counts previously read out of this registry, clamping every counter at zero.
+   *
+   * Used to hand a batch of counters over to a consumer that owns them from then on -- the commit-boundary
+   * drain for the record index lookup counters -- without discarding whatever arrived after they were read.
+   *
+   * Clamping is what distinguishes this from {@code add(name, -value)}. The registry can be emptied
+   * underneath a caller by an unrelated destructive scrape ({@link #getAllMetrics(boolean, boolean)} with
+   * {@code flush=true} clears every registry in the process), and an unbounded subtraction would then leave
+   * negative counters behind for good.
+   *
+   * The default is a best-effort read-modify-write. Implementations able to do this atomically should
+   * override it, and should drop counters that reach zero rather than leaving them at zero, so a registry
+   * nobody is writing to reads as empty.
+   *
+   * @param counts the counts to release, as returned by {@link #getAllCounts(boolean)}.
+   */
+  default void release(Map<String, Long> counts) {

Review Comment:
   Dead-but-harmless API surface in hudi-io, ~15 lines. It is part of the same hudi-io bundle as thread 24  (`getRegistry` contract, `NoOpRegistry`, `ExecutorMetricsContext`), which you chose to keep open without a reply. Consistent handling would be the same here; this is a nit, so let's not block on this.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863381658


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {

Review Comment:
   Carried to #19759 (section E) as a known limitation, and the PR body now states the SQL DML / StreamSync carry-over next to the DataSource one. Resolving on that basis; the one-line fix is still `dd4a48c` if you would rather take it here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863013178


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/DistributedRegistry.java:
##########
@@ -48,9 +51,17 @@ public String getName() {
   public void register(JavaSparkContext jsc) {
     if (!isRegistered()) {
       jsc.sc().register(this);
+      // Only when this call actually registers: stamping unconditionally would re-brand an accumulator
+      // bound to a dead context and mask the staleness this field exists to detect.
+      this.registeredAppId = jsc.sc().applicationId();
     }
   }
 
+  /** False when bound to a different (typically stopped) context, meaning it must be recreated. */
+  public boolean isRegisteredWith(JavaSparkContext jsc) {

Review Comment:
   The branch is already stacked for this: `ee961488` is the hardening on its own (`DistributedRegistry`, `getMetricRegistry`, `Registry` / `ExecutorMetricsContext` / `NoOpRegistry` and their four test classes, 811 lines, nothing RLI-specific), with `8c4d0c96` and `8a71f719` on top. It fixes `hoodie.metrics.executor.enable` for `HoodieWrapperFileSystem` on master regardless of where the counter design (name resolution, instance scoping) ends up, so could `ee961488` plus the guard and remover hunks from `05a5243` go out first as #19063's PR 1, with this PR rebased on it? That is one `rebase -i` rather than a restack, and it lets the fix land even if the counters take another round.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5426332461

   Followup items are all listed here: 
   https://github.com/apache/hudi/issues/19759


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863491594


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/DistributedRegistry.java:
##########
@@ -48,9 +51,17 @@ public String getName() {
   public void register(JavaSparkContext jsc) {
     if (!isRegistered()) {
       jsc.sc().register(this);
+      // Only when this call actually registers: stamping unconditionally would re-brand an accumulator
+      // bound to a dead context and mask the staleness this field exists to detect.
+      this.registeredAppId = jsc.sc().applicationId();
     }
   }
 
+  /** False when bound to a different (typically stopped) context, meaning it must be recreated. */
+  public boolean isRegisteredWith(JavaSparkContext jsc) {

Review Comment:
   Fine to land it together now that the rest has converged; noted on #19063 that gaps 2/7/8 arrive via this PR, with `ee961488` as the extractable fallback if it stalls. Resolving.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863543643


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metrics/TestDistributedRegistry.java:
##########
@@ -184,6 +187,131 @@ public void testAddMetricsParallel() {
     Assertions.assertEquals(finalExpectedSum, metricCounts.get(METRIC_1));
   }
 
+  @Test
+  public void testSetThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testSetOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: set() invoked on an executor must fail - it is non-commutative under accumulator merges.
+    // The UnsupportedOperationException thrown on the executor surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.set() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.set(METRIC_1, value);
+          return null;
+        }, 1));
+  }
+
+  @Test
+  public void testReleaseThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testReleaseOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: release() invoked on an executor must fail - clamping and eviction are order-dependent
+    // under accumulator merges. The UnsupportedOperationException surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.release() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.release(Collections.singletonMap(METRIC_1, (long) value));
+          return null;
+        }, 1));
+  }
+
+  /**
+   * Asserts the job failed because the executor-side guard fired, not for some unrelated reason such as a serialization error.
+   */
+  private static void assertFailsOnExecutorWith(String expectedMessage, Executable executable) {
+    SparkException thrown = Assertions.assertThrows(SparkException.class, executable);
+    StringBuilder chain = new StringBuilder();
+    for (Throwable t = thrown; t != null; t = t.getCause()) {
+      chain.append(t).append('\n');
+      if (t.getCause() == t) {
+        break;
+      }
+    }
+    Assertions.assertTrue(chain.toString().contains(expectedMessage),
+        "expected the executor-side guard to fail the job, got: " + chain);
+  }
+
+  @Test
+  public void testSetOnDriverSucceeds() {
+    // set() on the driver (no TaskContext) remains supported.
+    DistributedRegistry registry = new DistributedRegistry(REGISTRY_NAME + "_testSetOnDriver");
+    registry.set(METRIC_1, 42);
+    Assertions.assertEquals(42, registry.getAllCounts().get(METRIC_1));
+  }
+
+  @Test
+  public void testGetMetricRegistryReplacesNonDistributedRegistry() {

Review Comment:
   Branch removal tracked on #19759 (section A); the rename stands as asked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5419016562

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8a71f71978edc6f83d37711a582e516b743aa7e7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468) 
   * e62a131d0a4821a474fc78f037df1c82fcc1c559 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): make record index lookups observable via executor metrics [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5361903696

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f949227bee97d7b41d87d8c18ff73f6d15d7de17 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391) 
   * 70968d47931e1e491260eccd0eadd85e12a83202 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5372778614

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 457b1ac5c6ca7b2e46816e9c22d027bb498dbe41 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442) 
   * 8a71f71978edc6f83d37711a582e516b743aa7e7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5372843089

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 457b1ac5c6ca7b2e46816e9c22d027bb498dbe41 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442) 
   * 8a71f71978edc6f83d37711a582e516b743aa7e7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5275482560

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 369fbd591652f70be61c8c5fa19526955d41ba38 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5448810275

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     }, {
       "hash" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644",
       "triggerID" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bc8688a911937a4a0fef69304f182697141245d9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16651",
       "triggerID" : "bc8688a911937a4a0fef69304f182697141245d9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 695f74f6364193e7869f83b7eab88e9b637b555c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644) 
   * bc8688a911937a4a0fef69304f182697141245d9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16651) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5453097413

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     }, {
       "hash" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644",
       "triggerID" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bc8688a911937a4a0fef69304f182697141245d9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16651",
       "triggerID" : "bc8688a911937a4a0fef69304f182697141245d9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "29636dd2c30bd4c18507736bec602d938274ceb9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16669",
       "triggerID" : "29636dd2c30bd4c18507736bec602d938274ceb9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 29636dd2c30bd4c18507736bec602d938274ceb9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16669) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous merged PR #19575:
URL: https://github.com/apache/hudi/pull/19575


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5442822134

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 0ae8343d75734b1d9d89e04f1065337fc97faf9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612) 
   * 47a7be9e0ce5d6aec98bb7cd582222a5d754462f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5443594143

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 0ae8343d75734b1d9d89e04f1065337fc97faf9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612) 
   * 47a7be9e0ce5d6aec98bb7cd582222a5d754462f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638) 
   * fa7936408666ceed9010ee10a19b38fa04bec224 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5434687429

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * eadd7bbf6072d5c29c93f1650ae25f2b5f4de998 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606) 
   * 6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610) 
   * 0ae8343d75734b1d9d89e04f1065337fc97faf9b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5274911543

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f2ace685b8450e4466990371e11deba541aa992 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234) 
   * bb09a2d96ee77c288bc295a9195cfca97a81e3bd Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259) 
   * 369fbd591652f70be61c8c5fa19526955d41ba38 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5363858643

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3d9cb9c3c5e0631abdad072dd0a6706ac42d4521 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428) 
   * 30dbc00d04a21cd671b1e02b527f3b340f828008 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431) 
   * 2c37a62e311d57eb3636dc5f28af1739788c98f9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5365522625

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a061dbc0124d8ebc62defdf41d2d428842a6b16c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435) 
   * c3df340e3609cb1ad7f5f8abce5d207625887fa7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438) 
   * 457b1ac5c6ca7b2e46816e9c22d027bb498dbe41 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5422101690

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * bf51bb77e90f1cfa0fed51257b7dd1cc2052a830 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563) 
   * 0a9d4b412caeb2ef3dc19c1b16287265bd106fe7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5419676627

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e62a131d0a4821a474fc78f037df1c82fcc1c559 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555) 
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * 221b2b5bfec5eb6548462bd91fe85b3502f52637 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5423222242

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 0a9d4b412caeb2ef3dc19c1b16287265bd106fe7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5434305059

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * eadd7bbf6072d5c29c93f1650ae25f2b5f4de998 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606) 
   * 6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863538497


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {

Review Comment:
   Capture cost tracked on #19759 (section A).



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/RecordIndexMetricNames.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+/**
+ * Counter names for the record index lookup phase. Collection itself is generic -- see
+ * {@link ExecutorMetricRegistry} and {@link ExecutorMetrics}.
+ */
+public class RecordIndexMetricNames {
+
+  /** Scoping, prefix and reporter naming all live on the enum entry. */
+  public static final String REGISTRY_NAME = ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.registryName();
+
+  public static final String COMMIT_METADATA_PREFIX =
+      ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.commitMetadataPrefix();
+
+  // Counters are tagged by caller so dedupe traffic is distinguishable from tag-location traffic.
+  public static final String CALLER_TAG_LOCATION = "tag";
+  public static final String CALLER_DEDUPE = "dedupe";
+
+  public static final String KEY_COUNT = "lookup_record_index_key_count";

Review Comment:
   Dead constants and TODO tracked on #19759 (section C).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5434066059

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * eadd7bbf6072d5c29c93f1650ae25f2b5f4de998 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3877807797


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -146,8 +147,16 @@ public boolean commit(String instantTime, JavaRDD<WriteStatus> rawWriteStatuses,
       // when streaming writes are enabled, writeStatuses is a mix of data table write status and mdt write status
       List<HoodieWriteStat> dataTableHoodieWriteStats = slimWriteStatsList.stream().filter(entry -> !entry.isMetadataTable()).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
       List<HoodieWriteStat> partialMetadataTableWriteStats = slimWriteStatsList.stream().filter(entry -> entry.isMetadataTable).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
-      return commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds, extraPreCommitFunc,
-          false, Option.of(table));
+      boolean committed = commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds,
+          extraPreCommitFunc, false, Option.of(table));
+      // Publish only if this write tagged. requiresTagging is the same predicate BaseWriteHelper uses
+      // to decide whether to look up at all, so publishing mirrors collecting: an INSERT or a
+      // BULK_INSERT never collects, and must not publish counters an earlier write left behind.
+      if (committed && getOperationType() != null

Review Comment:
   Agree with moving this into the shared `postCommit` path; that would also cover writes landed through the public `commitStats` overloads.
   
   One caveat: `getIndex().requiresTagging(getOperationType())` is not equivalent to whether this write performed an RLI lookup. For an `INSERT` with `hoodie.datasource.write.insert.drop.duplicates=true` or a drop/fail duplicate policy, `HoodieSparkSqlWriter.handleInsertDuplicates` calls `DataSourceUtils.handleDuplicates` -> `SparkRDDReadClient.tagLocation`, so the instrumented RLI lookup fills the accumulator. `HoodieIndex.requiresTagging(INSERT)` is false, however, so this guard skips publication and the next tagging write clears those counters.
   
   Could the lifecycle track whether this write actually resolved/used the lookup registry, rather than infer that from operation type? A regression test with INSERT duplicate handling enabled would pin this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5434419169

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * eadd7bbf6072d5c29c93f1650ae25f2b5f4de998 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606) 
   * 6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3877789261


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -146,8 +147,16 @@ public boolean commit(String instantTime, JavaRDD<WriteStatus> rawWriteStatuses,
       // when streaming writes are enabled, writeStatuses is a mix of data table write status and mdt write status
       List<HoodieWriteStat> dataTableHoodieWriteStats = slimWriteStatsList.stream().filter(entry -> !entry.isMetadataTable()).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
       List<HoodieWriteStat> partialMetadataTableWriteStats = slimWriteStatsList.stream().filter(entry -> entry.isMetadataTable).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
-      return commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds, extraPreCommitFunc,
-          false, Option.of(table));
+      boolean committed = commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds,
+          extraPreCommitFunc, false, Option.of(table));
+      // Publish only if this write tagged. requiresTagging is the same predicate BaseWriteHelper uses
+      // to decide whether to look up at all, so publishing mirrors collecting: an INSERT or a
+      // BULK_INSERT never collects, and must not publish counters an earlier write left behind.
+      if (committed && getOperationType() != null

Review Comment:
   and it seems that `commitStats(` will never return false, it either return true or throws.
   
   also can we move the rli lookup metrics publish into the `postCommit` method? since that is where the metrics publish should be located.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -146,8 +147,16 @@ public boolean commit(String instantTime, JavaRDD<WriteStatus> rawWriteStatuses,
       // when streaming writes are enabled, writeStatuses is a mix of data table write status and mdt write status
       List<HoodieWriteStat> dataTableHoodieWriteStats = slimWriteStatsList.stream().filter(entry -> !entry.isMetadataTable()).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
       List<HoodieWriteStat> partialMetadataTableWriteStats = slimWriteStatsList.stream().filter(entry -> entry.isMetadataTable).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
-      return commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds, extraPreCommitFunc,
-          false, Option.of(table));
+      boolean committed = commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds,
+          extraPreCommitFunc, false, Option.of(table));
+      // Publish only if this write tagged. requiresTagging is the same predicate BaseWriteHelper uses
+      // to decide whether to look up at all, so publishing mirrors collecting: an INSERT or a
+      // BULK_INSERT never collects, and must not publish counters an earlier write left behind.
+      if (committed && getOperationType() != null

Review Comment:
   can we check if the rli lookup metrics is enabled in the very first place so that we avoid the unnecessary check, inside `RecordIndexLookupMetrics.publishAndRelease`, assumes the metircs is enabled, and if it is skipped, log some warning msg there.
   
   ```java
   if (<committed> && <rli lookup metrics enabled> && ...) {
     RecordIndexLookupMetrics.publishAndRelease(config, metrics);
   }
   ```
   
   and it seems that `commitStats(` will never return false, it either return true or throws.
   
   also can we move the rli lookup metrics publish into the `postCommit` method? since that is where the metrics publish should be located.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3877793543


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/PartitionedRecordIndexFileGroupLookupFunction.java:
##########
@@ -65,11 +77,17 @@ public Iterator<Tuple2<String, HoodieRecordGlobalLocation>> call(Iterator<Pair<S
       return Collections.emptyIterator();
     }
 
+    HoodieTimer shardTimer = HoodieTimer.start();

Review Comment:
   only starts the timer when lookupMetrics != null ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5434852757

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * eadd7bbf6072d5c29c93f1650ae25f2b5f4de998 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606) 
   * 6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610) 
   * 0ae8343d75734b1d9d89e04f1065337fc97faf9b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3840445764


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {
+    for (Drained drained : counters.drained) {
+      publishToReporter(drained, hoodieMetrics);
+      drained.registry.release(drained.counts);
+    }
+  }
+
+  /** Gauges, so each commit overwrites the previous value rather than accumulating. */
+  private static void publishToReporter(Drained drained, HoodieMetrics hoodieMetrics) {
+    if (hoodieMetrics == null || hoodieMetrics.getMetrics() == null) {
+      return;
+    }
+    String prefix = hoodieMetrics.getMetricsName(
+        drained.metricRegistry.metricAction(), drained.metricRegistry.metricQualifier());
+    hoodieMetrics.getMetrics().registerGauges(drained.counts, Option.ofNullable(prefix));
+  }
+
+  private static final class Drained {

Review Comment:
   Is there a better naming convention we can use for this class?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5362983984

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 70968d47931e1e491260eccd0eadd85e12a83202 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426) 
   * 3d9cb9c3c5e0631abdad072dd0a6706ac42d4521 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5365004587

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a061dbc0124d8ebc62defdf41d2d428842a6b16c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435) 
   * c3df340e3609cb1ad7f5f8abce5d207625887fa7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858562162


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {

Review Comment:
   this will be removed



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {
+    for (Drained drained : counters.drained) {
+      publishToReporter(drained, hoodieMetrics);
+      drained.registry.release(drained.counts);
+    }
+  }
+
+  /** Gauges, so each commit overwrites the previous value rather than accumulating. */
+  private static void publishToReporter(Drained drained, HoodieMetrics hoodieMetrics) {
+    if (hoodieMetrics == null || hoodieMetrics.getMetrics() == null) {
+      return;
+    }
+    String prefix = hoodieMetrics.getMetricsName(
+        drained.metricRegistry.metricAction(), drained.metricRegistry.metricQualifier());
+    hoodieMetrics.getMetrics().registerGauges(drained.counts, Option.ofNullable(prefix));
+  }
+
+  private static final class Drained {

Review Comment:
   this will be removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858308678


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricGroup.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.config.HoodieWriteConfig;
+
+/**
+ * One class of executor-collected metric. Implemented by {@link ExecutorMetricRegistry}, which enumerates
+ * the ones that ship; kept an interface so the collection machinery can be exercised against a group it
+ * does not know about.
+ */
+public interface ExecutorMetricGroup {

Review Comment:
   Ok for now we can remove the interface and make this simpler. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858618050


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {

Review Comment:
   Not sure i understand this comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5420576963

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * 221b2b5bfec5eb6548462bd91fe85b3502f52637 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556) 
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 308c956e2a5895f99806e8d2d242f318cf670c42 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560) 
   * bf51bb77e90f1cfa0fed51257b7dd1cc2052a830 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859017120


##########
hudi-common/src/test/java/org/apache/hudi/common/TestRegistryExecutorLookup.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common;
+
+import org.apache.hudi.common.metrics.ExecutorMetricsContext;
+import org.apache.hudi.common.metrics.LocalRegistry;
+import org.apache.hudi.common.metrics.Registry;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Characterizes how an executor reaches a driver-published registry by name. */
+public class TestRegistryExecutorLookup {

Review Comment:
   Fixed. Added an `@AfterEach` clearing `Registry.REGISTRY_MAP`, so the entries no longer outlive the class under `reuseForks`.



##########
hudi-common/src/main/java/org/apache/hudi/common/config/metrics/HoodieMetricsConfig.java:
##########
@@ -109,6 +109,17 @@ public class HoodieMetricsConfig extends HoodieConfig {
       .sinceVersion("0.13.0")
       .withDocumentation("Enable metrics for locking infra. Useful when operating in multiwriter mode");
 
+  public static final ConfigProperty<Boolean> RLI_LOOKUP_METRICS_ENABLE = ConfigProperty

Review Comment:
   Added `withRecordIndexLookupMetrics(boolean)` for parity with the siblings.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricGroup.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.config.HoodieWriteConfig;
+
+/**
+ * One class of executor-collected metric. Implemented by {@link ExecutorMetricRegistry}, which enumerates
+ * the ones that ship; kept an interface so the collection machinery can be exercised against a group it
+ * does not know about.
+ */
+public interface ExecutorMetricGroup {

Review Comment:
   Done. Removed `ExecutorMetricGroup`, `TestExecutorMetricsGenericity`, both test-only `groups` overloads and the unused `RecordIndexMetricNames.registryName` -- 236 lines. You were right that the 3-arg `resolveBundle` had no caller at all. The enum was already the extension point; an interface can come back when a second group lands and its shape is known.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -255,6 +256,12 @@ public boolean commitStats(String instantTime, TableWriteStats tableWriteStats,
       return true;
     }
     extraMetadata = updateExtraMetadata(extraMetadata);
+    // The commit is the boundary for the record index lookup counters. Snapshot them into this commit's
+    // metadata now, but only release them from the registry once the commit has actually landed --
+    // hooked here rather than in updateExtraMetadata because that is shared with table-service
+    // scheduling, which would otherwise consume the counters into a compaction or clustering plan.
+    ExecutorMetrics.DrainedCounters executorCounters =
+        ExecutorMetrics.snapshotIntoCommitMetadata(extraMetadata.get(), config);

Review Comment:
   Resolved by removing the sink. The counters no longer go into commit metadata at all -- the metrics reporter is the only destination now -- so nothing mutates `extraMetadata` and `runTableServicesInline` has nothing to copy forward. The misattribution is gone structurally rather than by copying the map.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {

Review Comment:
   Fixed. A commit that collected nothing now explicitly zeroes the gauge names this group has already published, so a scheduled reporter cannot re-emit the previous commit's numbers. Only names already published are touched, so a table that has never emitted stays absent rather than reporting a row of zeros.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5420043925

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e62a131d0a4821a474fc78f037df1c82fcc1c559 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555) 
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * 221b2b5bfec5eb6548462bd91fe85b3502f52637 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556) 
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 308c956e2a5895f99806e8d2d242f318cf670c42 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3862990025


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {
+    DISTRIBUTED_REGISTRY_MAP.remove(tableName.isEmpty() ? registryName : tableName + "." + registryName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {

Review Comment:
   Agreed on scoping it out. Since 16, 24, 39, 41 and the `Metrics.shutdown` scrape on the `ExecutorMetricRegistry:78` thread all defer to this same change and none of them has a tracking artifact yet, could you add one checklist line to #19063 listing them, so they are not lost when this merges?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5365595078

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c3df340e3609cb1ad7f5f8abce5d207625887fa7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438) 
   * 457b1ac5c6ca7b2e46816e9c22d027bb498dbe41 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5263029667

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5d3fa9e8ab489adcd8b266911cfaee7672ab8846 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227) 
   * 4f2ace685b8450e4466990371e11deba541aa992 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859017918


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {

Review Comment:
   Agreed on the direction, reached from the other end. Publishing now happens only after the commit lands, so an abandoned attempt publishes nothing, and the `AcrossFailedCommit` assertion is flipped to "the retry reports only its own lookups" as you suggested.
   
   Your parenthetical turned out to be the operative fact: with `hoodie.metrics.on=true` the carry-over does not hold anyway, because `Metrics.shutdown()` runs `Registry.getAllMetrics(true, true)` and clears every registry after each DataSource write. So on that path an abandoned attempt's counters are dropped rather than carried, which is the safer of the two failure modes. That is now stated in the PR description and asserted by the test rather than left implicit.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDReadClient.java:
##########
@@ -210,8 +212,16 @@ public JavaRDD<HoodieRecord<T>> filterExists(JavaRDD<HoodieRecord<T>> hoodieReco
    * @return Tagged RDD of Hoodie records
    */
   public JavaRDD<HoodieRecord<T>> tagLocation(JavaRDD<HoodieRecord<T>> hoodieRecords) throws HoodieIndexException {
-    return HoodieJavaRDD.getJavaRDD(
-        index.tagLocation(HoodieJavaRDD.of(hoodieRecords), context, hoodieTable));
+    // Lookups driven from the read client are dedupe traffic, not tag-location traffic. Label them so
+    // the two are distinguishable in the reported counters. Driver-side only: the label is
+    // captured when the lookup closure is built.
+    String previousCaller = RecordIndexLookupMetrics.setCaller(RecordIndexMetricNames.CALLER_DEDUPE);

Review Comment:
   Resolved by removing caller tagging from this PR entirely. You were right that it had no coverage, and the `checkExists` mislabelling you spotted confirms it was not carrying its weight as written. Tracked for restoration with the coverage it was missing, and the first question there is whether the dedupe path is reachable enough to be worth it -- `SparkRDDReadClient.tagLocation` has one production caller and insert-dedupe is off by default.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",

Review Comment:
   Moot now -- the counters no longer reach the timeline, so there is no permanent extra-metadata key to name. The reporter gauge names are `<table>.rli.lookup.<metric>`, which follows the existing `<table>.<action>.<metric>` convention used by `HoodieMetrics`.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRliLookupMetricsOnSparkSql.scala:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import org.apache.spark.sql.SaveMode
+import org.junit.jupiter.api.{Tag, Test}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue}
+
+/** Record level index lookup counters on the Spark SQL write path. */
+@Tag("functional")
+class TestRliLookupMetricsOnSparkSql extends RliLookupMetricsTestBase {
+
+  private val sqlTable = "rli_lookup_metrics_tbl"
+  private val numSeedRecords = 60
+
+  /**
+   * Seeds a table through the DataSource so the record index exists, then exposes it to SQL and applies
+   * the index settings as session configs -- index type is a write config, not a table property.
+   */
+  private def seedTableAndRegisterForSql(): Unit = {
+    doWriteAndValidateDataAndRecordIndex(rliOpts, INSERT_OPERATION_OPT_VAL, SaveMode.Overwrite,
+      validate = false, numInserts = numSeedRecords)
+
+    spark.sql(s"drop table if exists $sqlTable")
+    spark.sql(s"create table $sqlTable using hudi location '$basePath'")
+
+    spark.sql("set hoodie.write.lock.provider = org.apache.hudi.client.transaction.lock.InProcessLockProvider")
+    spark.sql(s"set ${HoodieMetadataConfig.ENABLE.key} = true")
+    spark.sql(s"set ${HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key} = ${!isPartitionedRli}")
+    spark.sql(s"set ${HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key} = $isPartitionedRli")
+    spark.sql(s"set ${HoodieIndexConfig.INDEX_TYPE.key} = " +
+      (if (isPartitionedRli) "RECORD_LEVEL_INDEX" else "GLOBAL_RECORD_LEVEL_INDEX"))
+
+    clearRliRegistry()
+  }
+
+  /**
+   * The default path. Optimized writes make UPDATE a prepped write, so no index lookup happens and no
+   * counters are produced. Documented behaviour, asserted so it cannot change unnoticed.
+   */
+  @Test
+  def testUpdateWithOptimizedWritesPerformsNoLookup(): Unit = {
+    seedTableAndRegisterForSql()
+    spark.sql(s"set ${SPARK_SQL_OPTIMIZED_WRITES.key} = true")
+
+    spark.sql(s"update $sqlTable set rider = 'rider-optimized'")
+
+    val counters = rliCountersFromLatestCommit()
+    report(s"Spark SQL UPDATE, optimized writes ON ($indexLabel) -- expected empty", counters)
+    assertTrue(counters.isEmpty,

Review Comment:
   Resolved -- that test class is no longer in the PR. Spark SQL turned out not to report these counters at all, so rather than land a test that passes vacuously I removed the class and filed apache/hudi#19740 with its full source attached.



##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metrics/TestExecutorMetricsGenericity.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.ExecutorMetricsContext;
+import org.apache.hudi.common.metrics.LocalRegistry;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The claim this feature exists to support is that adding a class of executor metric costs a declaration
+ * plus two lines at the emission site. These tests hold that claim to a measurement rather than an
+ * argument, by collecting a group the shipping code has never heard of.
+ */
+public class TestExecutorMetricsGenericity {
+
+  private static final String BASE_PATH = "file:///tmp/test_generic_metrics";
+
+  /** A class of metric added by a hypothetical future contributor: a name, a prefix, and no config. */
+  private static final ExecutorMetricGroup STORAGE_CALLS = new ExecutorMetricGroup() {
+    @Override
+    public String registryName() {
+      return "HoodieStorageCalls";
+    }
+
+    @Override
+    public String commitMetadataPrefix() {
+      return "hoodie.storage.calls.";
+    }
+
+    @Override
+    public String metricAction() {
+      return "storage";
+    }
+
+    @Override
+    public String metricQualifier() {
+      return "calls";
+    }
+
+    @Override
+    public boolean isEnabled(HoodieWriteConfig config) {
+      return true;
+    }
+
+    @Override
+    public String scopedName(String basePath) {
+      return registryName() + ".test";
+    }
+  };
+
+  private static HoodieWriteConfig config() {
+    return HoodieWriteConfig.newBuilder().withPath(BASE_PATH).forTable("generic_metrics_table").build();
+  }
+
+  private static Registry seed(HoodieWriteConfig cfg) {
+    Registry registry = new LocalRegistry(STORAGE_CALLS.scopedName(cfg.getBasePath()));
+    Registry.REGISTRY_MAP.put(

Review Comment:
   Resolved -- `TestExecutorMetricsGenericity` is deleted along with the abstraction it existed to exercise.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859020547


##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metrics/TestDistributedRegistry.java:
##########
@@ -184,6 +187,131 @@ public void testAddMetricsParallel() {
     Assertions.assertEquals(finalExpectedSum, metricCounts.get(METRIC_1));
   }
 
+  @Test
+  public void testSetThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testSetOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: set() invoked on an executor must fail - it is non-commutative under accumulator merges.
+    // The UnsupportedOperationException thrown on the executor surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.set() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.set(METRIC_1, value);
+          return null;
+        }, 1));
+  }
+
+  @Test
+  public void testReleaseThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testReleaseOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: release() invoked on an executor must fail - clamping and eviction are order-dependent
+    // under accumulator merges. The UnsupportedOperationException surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.release() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.release(Collections.singletonMap(METRIC_1, (long) value));
+          return null;
+        }, 1));
+  }
+
+  /**
+   * Asserts the job failed because the executor-side guard fired, not for some unrelated reason such as a serialization error.
+   */
+  private static void assertFailsOnExecutorWith(String expectedMessage, Executable executable) {

Review Comment:
   Resolved incidentally -- both tests were rewritten to assert behaviour rather than the exception, since the guards now warn and return instead of throwing. No message matching left.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859019676


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",
+      "rli",
+      "lookup",
+      HoodieWriteConfig::isRecordIndexLookupMetricsEnabled);
+
+  private final String registryName;
+  private final String commitMetadataPrefix;
+  private final String metricAction;
+  private final String metricQualifier;
+  private final Predicate<HoodieWriteConfig> enabled;
+
+  ExecutorMetricRegistry(String registryName, String commitMetadataPrefix, String metricAction,
+                         String metricQualifier, Predicate<HoodieWriteConfig> enabled) {
+    this.registryName = registryName;
+    this.commitMetadataPrefix = commitMetadataPrefix;
+    this.metricAction = metricAction;
+    this.metricQualifier = metricQualifier;
+    this.enabled = enabled;
+  }
+
+  /** The bare name emitting code passes to {@link Registry#getRegistry(String)}. */
+  @Override
+  public String registryName() {
+    return registryName;
+  }
+
+  @Override
+  public String commitMetadataPrefix() {
+    return commitMetadataPrefix;
+  }
+
+  @Override
+  public String metricAction() {
+    return metricAction;
+  }
+
+  @Override
+  public String metricQualifier() {
+    return metricQualifier;
+  }
+
+  /** Gating here, rather than in the drain, is what lets a new class of metric need no new config. */
+  @Override
+  public boolean isEnabled(HoodieWriteConfig config) {
+    return enabled.test(config);
+  }
+
+  /**
+   * Driver-side {@code REGISTRY_MAP} key. Table name alone is not an identity: two tables can share one
+   * and would then share a registry. Executors use the bare {@link #registryName()}.
+   */
+  @Override
+  public String scopedName(String basePath) {

Review Comment:
   Real, and I hit it independently from the other direction while debugging: `Metrics.shutdown()` running `Registry.getAllMetrics(true, true)` is also why an abandoned commit's counters cannot survive to the retry on the DataSource path.
   
   In practice the republish does not fire today, because release-by-subtraction leaves the registry empty by the time teardown runs -- so it needs leftovers to be visible at all. Deferring the digest-free name / scrape exclusion to the registry-lifetime follow-up, since the digest only exists to disambiguate inside the process-wide map and both go away together.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",
+      "rli",
+      "lookup",
+      HoodieWriteConfig::isRecordIndexLookupMetricsEnabled);
+
+  private final String registryName;
+  private final String commitMetadataPrefix;
+  private final String metricAction;
+  private final String metricQualifier;
+  private final Predicate<HoodieWriteConfig> enabled;
+
+  ExecutorMetricRegistry(String registryName, String commitMetadataPrefix, String metricAction,
+                         String metricQualifier, Predicate<HoodieWriteConfig> enabled) {
+    this.registryName = registryName;
+    this.commitMetadataPrefix = commitMetadataPrefix;
+    this.metricAction = metricAction;
+    this.metricQualifier = metricQualifier;
+    this.enabled = enabled;
+  }
+
+  /** The bare name emitting code passes to {@link Registry#getRegistry(String)}. */
+  @Override
+  public String registryName() {
+    return registryName;
+  }
+
+  @Override
+  public String commitMetadataPrefix() {
+    return commitMetadataPrefix;
+  }
+
+  @Override
+  public String metricAction() {
+    return metricAction;
+  }
+
+  @Override
+  public String metricQualifier() {
+    return metricQualifier;
+  }
+
+  /** Gating here, rather than in the drain, is what lets a new class of metric need no new config. */
+  @Override
+  public boolean isEnabled(HoodieWriteConfig config) {
+    return enabled.test(config);
+  }
+
+  /**
+   * Driver-side {@code REGISTRY_MAP} key. Table name alone is not an identity: two tables can share one
+   * and would then share a registry. Executors use the bare {@link #registryName()}.
+   */
+  @Override
+  public String scopedName(String basePath) {

Review Comment:
   Related to this, and worth recording: the digest in this name was not only ugly, it was wrong. Digesting the raw base path meant one table hashed two ways -- scheme-qualified from the Spark SQL catalog location, bare from the DataSource -- so that write path silently reported nothing at all. Now fixed by digesting authority + path.
   
   That does not change your point. The digest exists only because the map is process-wide, and both it and the common-scrape republish go away together when the registry becomes instance-scoped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5421581346

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * bf51bb77e90f1cfa0fed51257b7dd1cc2052a830 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563) 
   * 0a9d4b412caeb2ef3dc19c1b16287265bd106fe7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3877767498


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -146,8 +147,16 @@ public boolean commit(String instantTime, JavaRDD<WriteStatus> rawWriteStatuses,
       // when streaming writes are enabled, writeStatuses is a mix of data table write status and mdt write status
       List<HoodieWriteStat> dataTableHoodieWriteStats = slimWriteStatsList.stream().filter(entry -> !entry.isMetadataTable()).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
       List<HoodieWriteStat> partialMetadataTableWriteStats = slimWriteStatsList.stream().filter(entry -> entry.isMetadataTable).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
-      return commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds, extraPreCommitFunc,
-          false, Option.of(table));
+      boolean committed = commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds,
+          extraPreCommitFunc, false, Option.of(table));
+      // Publish only if this write tagged. requiresTagging is the same predicate BaseWriteHelper uses
+      // to decide whether to look up at all, so publishing mirrors collecting: an INSERT or a
+      // BULK_INSERT never collects, and must not publish counters an earlier write left behind.
+      if (committed && getOperationType() != null

Review Comment:
   can we check if the rli lookup metrics is enabled in the very first place so that we avoid the unnecessary check, inside `RecordIndexLookupMetrics.publishAndRelease`, assumes the metircs is enabled, and if it is skipped, log some warning msg there.
   
   ```java
   if (<committed> && <rli lookup metrics enabled> && ...) {
     RecordIndexLookupMetrics.publishAndRelease(config, metrics);
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3877791863


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/PartitionedRecordIndexFileGroupLookupFunction.java:
##########
@@ -44,9 +47,18 @@ public class PartitionedRecordIndexFileGroupLookupFunction
     implements PairFlatMapFunction<Iterator<Pair<String, String>>, String, HoodieRecordGlobalLocation> {
 
   private final HoodieTableMetadata metadataTable;
+  // Null when no counters should be collected; see RecordIndexLookupMetrics#resolveRegistry.
+  private final Registry lookupMetrics;
 
+  /** Uninstrumented, for the query-side read path. */
   public PartitionedRecordIndexFileGroupLookupFunction(HoodieTableMetadata metadataTable) {
+    this(metadataTable, null);
+  }
+
+  public PartitionedRecordIndexFileGroupLookupFunction(HoodieTableMetadata metadataTable,
+                                                       Registry lookupMetrics) {

Review Comment:
   passing around Option instead of null



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3874972540


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  /**
+   * The registry a lookup task collects into, or null when nothing should be collected. Captured in the
+   * lookup closure and passed to {@link #recordShardLookup}, so delivery is by closure capture rather
+   * than by name.
+   *
+   * <p>Requires the reporter to be on as well: with {@code hoodie.metrics.on} off there is nowhere to
+   * publish, and collecting would register an accumulator and scan every shard for nothing.
+   */
+  public static Registry resolveRegistry(HoodieEngineContext context, HoodieWriteConfig config) {
+    if (!config.isMetricsOn() || !ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.isEnabled(config)) {
+      return null;
+    }
+    // TBL_NAME has no default and Builder.validate() only requires BASE_PATH, so a config built without
+    // forTable() reaches here with a null name. getMetricRegistry dereferences it immediately.
+    String tableName = config.getTableName();
+    if (tableName == null || tableName.isEmpty()) {
+      return null;
+    }
+    Registry registry = context.getMetricRegistry(tableName,
+        ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.scopedName(config.getBasePath()));
+    // Only the accumulator-backed registry aggregates back to the driver. A LocalRegistry here would
+    // collect on the executor and be dropped on the floor, so report nothing instead.
+    if (!(registry instanceof DistributedRegistry)) {
+      return null;
+    }
+    // Runs on the driver before the closure ships, so anything held now predates this write. Publishing
+    // releases what it reports, which leaves a non-empty registry only after an attempt that never
+    // committed. Paths that tear metrics down between writes lose it anyway; Spark SQL DML and StreamSync
+    // do not, and this commit must not report work it did not do.
+    registry.clear();

Review Comment:
   Confirmed, and the destructive `clear()` is mine, so this is fair.
   
   One scoping point that I think bounds it. Every piece of state involved is per-JVM:
   
   - `Registry.REGISTRY_MAP` is an interface field, so implicitly `public static final`
   - `HoodieSparkEngineContext.DISTRIBUTED_REGISTRY_MAP` is `private static final`
   - the counters are an instance field on the `DistributedRegistry`, and merging goes through the task-result envelope of the `SparkContext` it was registered with
   
   Nothing is persisted outside the process either. The commit-metadata sink was removed earlier in this PR, so the only outbound path is `registerGauges`.
   
   So two write clients in one JVM do share a bucket and can clear and publish each other's counts, exactly as you describe. Two separate applications writing the same table cannot: separate drivers, separate static maps, separate accumulators, and an executor in one application has no route to the other's accumulator.
   
   On disabling for multiwriter: the gate would have to key off `WriteConcurrencyMode.supportsMultiWriter()`, which cannot distinguish separate-application multiwriter from in-process. It would switch the feature off for deployments that are not affected, including the one that asked for it, while the case it guards is the narrower in-process one.
   
   On scoping: instant-based scoping is not reachable from where the registry is resolved. `lookupRecords(records, context, hoodieTable, fileGroupSize)` carries no instant, and `HoodieTable` takes it as a parameter rather than holding it. A per-writer key is reachable in about eight lines, by stamping a UUID into the write config at client construction and including it in the registry key. The cost is that the key can then only be reconstructed from a live client config, which breaks the test helper that seeds the abandoned-attempt regression case.
   
   Proposal: take this as a follow-up scoped on whether in-process concurrent writers are actually in use, and document the limitation on the config in the meantime. It sits naturally with the registry-lifetime work already tracked in #19759, which removes the process-wide map and makes per-writer scoping fall out of it.
   
   Do you know whether the requesting deployment runs concurrent write clients inside one JVM? If it does, the eight-line key is worth doing now rather than as a follow-up.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3870637479


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/SparkMetadataTableGlobalRecordLevelIndex.java:
##########
@@ -181,21 +185,27 @@ public boolean isImplicitWithStorage() {
    */
   private static class RecordIndexFileGroupLookupFunction implements PairFlatMapFunction<Iterator<String>, String, HoodieRecordGlobalLocation> {
     private final HoodieTable hoodieTable;
+    // Null when no counters should be collected; see RecordIndexLookupMetrics#resolveRegistry.
+    private final Registry lookupMetrics;
 
-    public RecordIndexFileGroupLookupFunction(HoodieTable hoodieTable) {
+    public RecordIndexFileGroupLookupFunction(HoodieTable hoodieTable, Registry lookupMetrics) {
       this.hoodieTable = hoodieTable;
+      this.lookupMetrics = lookupMetrics;
     }
 
     @Override
     public Iterator<Tuple2<String, HoodieRecordGlobalLocation>> call(Iterator<String> recordKeyIterator) {
       List<String> keysToLookup = new ArrayList<>();
       recordKeyIterator.forEachRemaining(keysToLookup::add);
 
+      HoodieTimer shardTimer = HoodieTimer.start();
       // recordIndexInfo object only contains records that are present in record_index.
       HoodiePairData<String, HoodieRecordGlobalLocation> recordIndexData =
           hoodieTable.getTableMetadata().readRecordIndexLocationsWithKeys(HoodieListData.eager(keysToLookup));
       try {
         List<Pair<String, HoodieRecordGlobalLocation>> recordIndexInfo = HoodieDataUtils.dedupeAndCollectAsList(recordIndexData);
+        RecordIndexLookupMetrics.recordShardLookup(lookupMetrics, keysToLookup,
+            recordIndexInfo.stream().map(Pair::getKey).collect(Collectors.toSet()), shardTimer.endTimer());

Review Comment:
   [P1] Avoid disabled-path hit-set allocation
   
   `lookupMetrics` is null by default, but Java evaluates this argument before `recordShardLookup` can take its null fast path. Every global-RLI shard therefore still traverses all hits and allocates an O(hits) `HashSet` when the feature is disabled. For large high-hit upserts this makes an opt-in metric a default-path CPU/memory regression. Could we guard this block with `if (lookupMetrics != null)` or defer construction of the set into the helper?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -146,8 +147,14 @@ public boolean commit(String instantTime, JavaRDD<WriteStatus> rawWriteStatuses,
       // when streaming writes are enabled, writeStatuses is a mix of data table write status and mdt write status
       List<HoodieWriteStat> dataTableHoodieWriteStats = slimWriteStatsList.stream().filter(entry -> !entry.isMetadataTable()).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
       List<HoodieWriteStat> partialMetadataTableWriteStats = slimWriteStatsList.stream().filter(entry -> entry.isMetadataTable).map(SlimWriteStats::getWriteStat).collect(Collectors.toList());
-      return commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds, extraPreCommitFunc,
-          false, Option.of(table));
+      boolean committed = commitStats(instantTime, new TableWriteStats(dataTableHoodieWriteStats, partialMetadataTableWriteStats), extraMetadata, commitActionType, partitionToReplacedFileIds,
+          extraPreCommitFunc, false, Option.of(table));
+      if (committed) {
+        // The commit landed, so the counters the executors collected for it can be reported and
+        // released. A commit that never lands publishes nothing.
+        ExecutorMetrics.publishAndRelease(config, metrics);

Review Comment:
   [P2] Publish from the actual successful-commit boundary
   
   This hook runs only through `SparkRDDWriteClient.commit`, but the inherited public `commitStats` overloads can also land a write directly. A manual RLI upsert committed through `commitStats` leaves its counters unpublished and unreleased; a following insert with no lookup can then publish those old counts as its own. Could publication be attached to the shared successful `commitStats` boundary, with a direct-`commitStats` regression test, so all supported commit entry points have the same lifecycle?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5447021179

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     }, {
       "hash" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * fa7936408666ceed9010ee10a19b38fa04bec224 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639) 
   * 695f74f6364193e7869f83b7eab88e9b637b555c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3877808409


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,47 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistry(String tableName, String registryName) {
+    String prefixedName = tableName.isEmpty() ? registryName : tableName + "." + registryName;
+    DISTRIBUTED_REGISTRY_MAP.remove(prefixedName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+    // setRegistries also indexes under the empty table name, so evicting only the table-scoped key leaves
+    // the accumulator reachable under ::<table>.<registry>.
+    Registry.REGISTRY_MAP.remove(Registry.makeKey("", prefixedName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {
     final String prefixedName = tableName.isEmpty() ? registryName : tableName + "." + registryName;
-    return DISTRIBUTED_REGISTRY_MAP.computeIfAbsent(prefixedName, key -> {
+    // Both maps are process-wide statics that outlive any SparkContext, so the staleness check and the
+    // recreation have to be atomic: otherwise one caller can evict the registry another caller just
+    // created, leaving two live accumulators for one metric name while reporting only ever reads the
+    // one still in the map.
+    return DISTRIBUTED_REGISTRY_MAP.compute(prefixedName, (key, cached) -> {
+      if (cached instanceof DistributedRegistry && ((DistributedRegistry) cached).isRegisteredWith(javaSparkContext)) {
+        return cached;
+      }
+      // Nothing usable cached, or the cached accumulator is bound to a SparkContext that is no longer
+      // live (a restart in the same JVM: shells, notebooks, Spark Connect). Drop the shared-map entry
+      // first, since getRegistryOfClass() would otherwise hand back that same stale instance.
+      final String sharedKey = Registry.makeKey(tableName, registryName);
+      Registry.REGISTRY_MAP.remove(sharedKey);

Review Comment:
   I would go one step further and make the distributed-registry map owned by the `HoodieSparkEngineContext` instance instead of adding another override mode to the process-wide `Registry.REGISTRY_MAP`. That would hide the registry details here while also removing the base-path digest, stale-SparkContext replacement branch, test-only eviction helper, and the duplicate empty-table alias installed by `setRegistries`.
   
   It also creates a natural place to introduce per-writer or per-write ownership. With the current table-scoped static accumulator, two write clients in the same JVM can still clear and publish each other counters. This is the same lifetime simplification tracked in #19759, but it looks structurally cleaner than making `getRegistryOfClass` support replacement.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3879042453


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * The record index lookup counters, end to end: what they are called, where they are collected, and how
+ * they reach the reporter once a commit lands.
+ *
+ * <p>The numbers are only knowable on executors, because a lookup returns its hits and a miss produces
+ * no output row at all. So the driver resolves an accumulator-backed {@link DistributedRegistry}, the
+ * lookup closure captures it, tasks add into their own copy, and Spark merges those copies home.
+ */
+public class RecordIndexLookupMetrics {
+
+  private static final Logger LOG = LoggerFactory.getLogger(RecordIndexLookupMetrics.class);
+
+  /** The registry name, as reported. Keying is done by {@link #registryKey}. */
+  public static final String REGISTRY_NAME = "HoodieRecordIndexLookup";
+
+  /** Reporter naming, as passed to {@code HoodieMetrics.getMetricsName}. */
+  public static final String METRIC_ACTION = "rli";
+  public static final String METRIC_QUALIFIER = "lookup";
+
+  /** Counts records, not distinct keys: a batch repeating a key contributes once per record, which is
+   * what keeps {@code hits + misses == key_count} exact. */
+  public static final String KEY_COUNT = "lookup_record_index_key_count";
+  public static final String KEY_HIT_COUNT = "lookup_record_index_key_hit_count";
+  public static final String KEY_MISS_COUNT = "lookup_record_index_key_miss_count";
+  public static final String SHARDS_READ = "lookup_record_index_shards_read";
+  /**
+   * Wall-clock spent in the shard read, summed across shards rather than averaged because shards are read
+   * in parallel: the value is per-commit read effort, and dividing by {@link #SHARDS_READ} gives a mean.
+   *
+   * <p>Distinct from {@code index.lookup.duration} published by {@code HoodieMetrics.updateIndexMetrics},
+   * which is driver wall-clock for the whole {@code tagLocation} including scheduling. Comparing the two
+   * shows how much of a lookup was actually spent reading the index.
+   */
+  public static final String LOOKUP_TIME = "lookup_record_index_time";
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  /**
+   * The registry a lookup task collects into, or null when nothing should be collected. Captured in the
+   * lookup closure and passed to {@link #recordShardLookup}, so delivery is by closure capture rather
+   * than by name.
+   *
+   * <p>Requires the reporter to be on as well: with {@code hoodie.metrics.on} off there is nowhere to
+   * publish, and collecting would register an accumulator and scan every shard for nothing.
+   */
+  public static Option<Registry> resolveRegistry(HoodieEngineContext context, HoodieWriteConfig config) {
+    if (!config.isMetricsOn() || !config.isRecordIndexLookupMetricsEnabled()
+        || !(context instanceof HoodieSparkEngineContext)) {
+      return Option.empty();
+    }
+    DistributedRegistry registry = ((HoodieSparkEngineContext) context)
+        .getOrCreateOwnedRegistry(registryKey(config.getBasePath()), REGISTRY_NAME);
+    // Anything held predates this write: the drain removes the entry as it publishes, so a surviving
+    // one belongs to an attempt that never committed.
+    registry.clear();
+    return Option.of(registry);
+  }
+
+  /**
+   * Records one shard's lookup outcome. Counts records rather than distinct keys, so
+   * {@code hits + misses == records_looked_up} holds when a batch repeats a key. Membership is tested
+   * against the found set, bounded by the hit count, not the asked-about set, bounded by shard size.
+   *
+   * <p>Callers must skip the call entirely when the registry is null. Building the found set is
+   * O(hits), and an argument is evaluated before this method can take a null fast path, so guarding
+   * here would not keep that cost off the disabled path.
+   *
+   * @param registry     where to collect, never null
+   * @param keysLookedUp every record key routed to this shard
+   * @param foundKeys    the subset present in the index
+   * @param elapsedMs    wall-clock spent reading this shard
+   */
+  public static void recordShardLookup(Registry registry, Collection<String> keysLookedUp,
+                                       Collection<String> foundKeys, long elapsedMs) {
+    if (keysLookedUp.isEmpty()) {
+      return;
+    }
+    Set<String> found = foundKeys instanceof Set ? (Set<String>) foundKeys : new HashSet<>(foundKeys);
+    long records = keysLookedUp.size();
+    long hits = found.isEmpty() ? 0L : keysLookedUp.stream().filter(found::contains).count();
+    registry.add(KEY_COUNT, records);
+    registry.add(KEY_HIT_COUNT, hits);
+    registry.add(KEY_MISS_COUNT, records - hits);
+    registry.increment(SHARDS_READ);
+    registry.add(LOOKUP_TIME, elapsedMs);
+  }
+
+  /**
+   * Reports the counters and releases them, so the next commit reports only its own work. Called once
+   * the commit has landed, so a commit that never lands publishes nothing.
+   *
+   * <p>Release subtracts what was reported rather than clearing, so a straggler task whose update lands
+   * mid-publish carries into the next commit instead of being dropped. An all-zero registry is skipped.
+   */
+  public static void publishAndRelease(HoodieEngineContext context, HoodieWriteConfig config,
+                                       HoodieMetrics hoodieMetrics) {
+    try {
+      publish(context, config, hoodieMetrics);
+    } catch (Exception e) {
+      // This runs after the commit has landed. Reporting is not worth failing a completed write over.
+      LOG.warn("Failed to publish record index lookup metrics; the commit is unaffected.", e);
+    }
+  }
+
+  private static void publish(HoodieEngineContext context, HoodieWriteConfig config,
+                              HoodieMetrics hoodieMetrics) {
+    if (!config.isRecordIndexLookupMetricsEnabled()) {
+      // Callers gate before reaching here, so this is a wiring mistake rather than normal flow.
+      LOG.warn("Record index lookup metrics drain reached with the feature disabled; nothing published.");
+      return;
+    }
+    if (!(context instanceof HoodieSparkEngineContext)) {
+      return;
+    }
+    // Removing is what makes attribution work. An entry exists only because a lookup on this write
+    // created it, so its absence means this write looked nothing up and has nothing of its own to
+    // report -- which is the case an operation type cannot tell apart, since an insert that drops
+    // duplicates tags through SparkRDDReadClient without requiring tagging.
+    Option<DistributedRegistry> taken =
+        ((HoodieSparkEngineContext) context).removeOwnedRegistry(registryKey(config.getBasePath()));

Review Comment:
   [P1] Reuse the committing client context for INSERT duplicate lookup
   
   This drain assumes the lookup and commit use the same HoodieSparkEngineContext. However, INSERT duplicate handling currently goes through HoodieSparkSqlWriter.handleInsertDuplicates to DataSourceUtils.resolveDuplicates, which constructs a new HoodieSparkEngineContext. The lookup registry therefore belongs to that throwaway context, while postCommit removes from the write client context here; taken is empty and the INSERT commit publishes no lookup counters. Please pass client.getEngineContext and client.getConfig into duplicate resolution instead, and add an INSERT_DROP_DUPS regression test that asserts the counters reach the reporter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3853273775


##########
hudi-common/src/main/java/org/apache/hudi/common/config/metrics/HoodieMetricsConfig.java:
##########
@@ -109,6 +109,17 @@ public class HoodieMetricsConfig extends HoodieConfig {
       .sinceVersion("0.13.0")
       .withDocumentation("Enable metrics for locking infra. Useful when operating in multiwriter mode");
 
+  public static final ConfigProperty<Boolean> RLI_LOOKUP_METRICS_ENABLE = ConfigProperty
+      .key(METRIC_PREFIX + ".rli.lookup.enable")
+      .defaultValue(true)

Review Comment:
   Two precedents for the default. `EMBED_ENGINE_PROPERTIES_IN_COMMIT_METADATA` (`CommitMetadataProperties:80-88`) defaults to `false` "since these add per-commit growth to the timeline", and the sibling `LOCK_METRICS_ENABLE` (L99-108) inherits from `hoodie.metrics.on` via `withInferFunction` rather than standing alone. Default-on here also makes the RLI registry the first thing in `DISTRIBUTED_REGISTRY_MAP` without opt-in (see the `resolveBundle` comment). Could the default be `false` for the first release, or at least infer from `TURN_METRICS_ON`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863066845


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",
+      "rli",
+      "lookup",
+      HoodieWriteConfig::isRecordIndexLookupMetricsEnabled);
+
+  private final String registryName;
+  private final String commitMetadataPrefix;
+  private final String metricAction;
+  private final String metricQualifier;
+  private final Predicate<HoodieWriteConfig> enabled;
+
+  ExecutorMetricRegistry(String registryName, String commitMetadataPrefix, String metricAction,
+                         String metricQualifier, Predicate<HoodieWriteConfig> enabled) {
+    this.registryName = registryName;
+    this.commitMetadataPrefix = commitMetadataPrefix;
+    this.metricAction = metricAction;
+    this.metricQualifier = metricQualifier;
+    this.enabled = enabled;
+  }
+
+  /** The bare name emitting code passes to {@link Registry#getRegistry(String)}. */
+  @Override
+  public String registryName() {
+    return registryName;
+  }
+
+  @Override
+  public String commitMetadataPrefix() {
+    return commitMetadataPrefix;
+  }
+
+  @Override
+  public String metricAction() {
+    return metricAction;
+  }
+
+  @Override
+  public String metricQualifier() {
+    return metricQualifier;
+  }
+
+  /** Gating here, rather than in the drain, is what lets a new class of metric need no new config. */
+  @Override
+  public boolean isEnabled(HoodieWriteConfig config) {
+    return enabled.test(config);
+  }
+
+  /**
+   * Driver-side {@code REGISTRY_MAP} key. Table name alone is not an identity: two tables can share one
+   * and would then share a registry. Executors use the bare {@link #registryName()}.
+   */
+  @Override
+  public String scopedName(String basePath) {

Review Comment:
   Spelled out: `Metrics.shutdown()` calls `registerHoodieCommonMetrics()`, which runs `Registry.getAllMetrics(true, true)` over every entry in `REGISTRY_MAP`, registers what it finds as gauges, then clears. The common prefix is applied only to registry names without a `.`, and this one's name is `<table>.HoodieRecordIndexLookup.<12hex>`, so any counters still in the registry at shutdown (the failed-commit case in `TestRliLookupMetricsAcrossFailedCommit`, or a straggler) are emitted as `<prefix>.<table>.HoodieRecordIndexLookup.<12hex>.lookup_record_index_key_count`, one series per base path. `rliCountersFromLatestCommit` filters on `.rli.lookup.` so the test cannot see it; the `report()` dump after the failed commit should. It goes away with an instance-scoped registry, so folding it into that follow-up is fine -- could the PR body's "dropped rather than double-counted" sentence say they are published under that name instead, since that is what a pushgateway user will see?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859134744


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRliLookupMetricsOnSparkSql.scala:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import org.apache.spark.sql.SaveMode
+import org.junit.jupiter.api.{Tag, Test}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue}
+
+/** Record level index lookup counters on the Spark SQL write path. */
+@Tag("functional")
+class TestRliLookupMetricsOnSparkSql extends RliLookupMetricsTestBase {
+
+  private val sqlTable = "rli_lookup_metrics_tbl"
+  private val numSeedRecords = 60
+
+  /**
+   * Seeds a table through the DataSource so the record index exists, then exposes it to SQL and applies
+   * the index settings as session configs -- index type is a write config, not a table property.
+   */
+  private def seedTableAndRegisterForSql(): Unit = {
+    doWriteAndValidateDataAndRecordIndex(rliOpts, INSERT_OPERATION_OPT_VAL, SaveMode.Overwrite,
+      validate = false, numInserts = numSeedRecords)
+
+    spark.sql(s"drop table if exists $sqlTable")
+    spark.sql(s"create table $sqlTable using hudi location '$basePath'")
+
+    spark.sql("set hoodie.write.lock.provider = org.apache.hudi.client.transaction.lock.InProcessLockProvider")
+    spark.sql(s"set ${HoodieMetadataConfig.ENABLE.key} = true")
+    spark.sql(s"set ${HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key} = ${!isPartitionedRli}")
+    spark.sql(s"set ${HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key} = $isPartitionedRli")
+    spark.sql(s"set ${HoodieIndexConfig.INDEX_TYPE.key} = " +
+      (if (isPartitionedRli) "RECORD_LEVEL_INDEX" else "GLOBAL_RECORD_LEVEL_INDEX"))
+
+    clearRliRegistry()
+  }
+
+  /**
+   * The default path. Optimized writes make UPDATE a prepped write, so no index lookup happens and no
+   * counters are produced. Documented behaviour, asserted so it cannot change unnoticed.
+   */
+  @Test
+  def testUpdateWithOptimizedWritesPerformsNoLookup(): Unit = {
+    seedTableAndRegisterForSql()
+    spark.sql(s"set ${SPARK_SQL_OPTIMIZED_WRITES.key} = true")
+
+    spark.sql(s"update $sqlTable set rider = 'rider-optimized'")
+
+    val counters = rliCountersFromLatestCommit()
+    report(s"Spark SQL UPDATE, optimized writes ON ($indexLabel) -- expected empty", counters)
+    assertTrue(counters.isEmpty,

Review Comment:
   Reversed -- the class is back in the PR, and your point stands in it. What looked like a property of the Spark SQL write path was a bug in the registry key: it digests the raw base path, and SQL builds its config from the catalog location (scheme-qualified) while the DataSource passes the bare path, so one table got two keys and the drain looked under the wrong one. Fixed by digesting authority + path. The class now passes 6/6 including the non-prepped UPDATE and MERGE INTO cases.
   
   Your original comment here is addressed too: the prepped-UPDATE test is no longer a vacuous pass, because the sibling cases in the same class now prove the feature works on this path.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {
+    DISTRIBUTED_REGISTRY_MAP.remove(tableName.isEmpty() ? registryName : tableName + "." + registryName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {

Review Comment:
   One more piece of evidence for this, found after the fact. The base-path digest was not just awkward, it was silently wrong: it digests the raw string, so a table reached through a scheme-qualified path (Spark SQL, via the catalog location) hashed differently from the same table reached through a bare path (DataSource), and the drain looked under a key nothing had registered. Spark SQL reported no counters at all as a result.
   
   Fixed in place by digesting authority + path. But the digest only exists to disambiguate inside the process-wide map, so an instance-scoped registry removes the failure mode rather than correcting it -- which strengthens the case for the follow-up you are describing here.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",
+      "rli",
+      "lookup",
+      HoodieWriteConfig::isRecordIndexLookupMetricsEnabled);
+
+  private final String registryName;
+  private final String commitMetadataPrefix;
+  private final String metricAction;
+  private final String metricQualifier;
+  private final Predicate<HoodieWriteConfig> enabled;
+
+  ExecutorMetricRegistry(String registryName, String commitMetadataPrefix, String metricAction,
+                         String metricQualifier, Predicate<HoodieWriteConfig> enabled) {
+    this.registryName = registryName;
+    this.commitMetadataPrefix = commitMetadataPrefix;
+    this.metricAction = metricAction;
+    this.metricQualifier = metricQualifier;
+    this.enabled = enabled;
+  }
+
+  /** The bare name emitting code passes to {@link Registry#getRegistry(String)}. */
+  @Override
+  public String registryName() {
+    return registryName;
+  }
+
+  @Override
+  public String commitMetadataPrefix() {
+    return commitMetadataPrefix;
+  }
+
+  @Override
+  public String metricAction() {
+    return metricAction;
+  }
+
+  @Override
+  public String metricQualifier() {
+    return metricQualifier;
+  }
+
+  /** Gating here, rather than in the drain, is what lets a new class of metric need no new config. */
+  @Override
+  public boolean isEnabled(HoodieWriteConfig config) {
+    return enabled.test(config);
+  }
+
+  /**
+   * Driver-side {@code REGISTRY_MAP} key. Table name alone is not an identity: two tables can share one
+   * and would then share a registry. Executors use the bare {@link #registryName()}.
+   */
+  @Override
+  public String scopedName(String basePath) {

Review Comment:
   Related to this, and worth recording: the digest in this name was not only ugly, it was wrong. Digesting the raw base path meant one table hashed two ways -- scheme-qualified from the Spark SQL catalog location, bare from the DataSource -- so that write path silently reported nothing at all. Now fixed by digesting authority + path.
   
   That does not change your point. The digest exists only because the map is process-wide, and both it and the common-scrape republish go away together when the registry becomes instance-scoped.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858308678


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricGroup.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.config.HoodieWriteConfig;
+
+/**
+ * One class of executor-collected metric. Implemented by {@link ExecutorMetricRegistry}, which enumerates
+ * the ones that ship; kept an interface so the collection machinery can be exercised against a group it
+ * does not know about.
+ */
+public interface ExecutorMetricGroup {

Review Comment:
   Ok for now we remove the interface and make this simpler. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5421065611

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * bf51bb77e90f1cfa0fed51257b7dd1cc2052a830 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863300953


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {
+    DISTRIBUTED_REGISTRY_MAP.remove(tableName.isEmpty() ? registryName : tableName + "." + registryName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {

Review Comment:
   Filed as #19759 with the deferred items listed; resolving on that basis.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = context.getMetricRegistry(config.getTableName(),
+          metricRegistry.scopedName(config.getBasePath()));
+      // Only the accumulator-backed registry aggregates back to the driver, so anything else is left out
+      // rather than bound: a bound LocalRegistry would collect on the executor and be dropped on the floor,
+      // whereas leaving it out makes the lookup resolve to a no-op that reports nothing.
+      if (registry instanceof DistributedRegistry) {
+        bundle.put(metricRegistry.registryName(), registry);
+      }
+    }
+    return bundle.isEmpty() ? Collections.emptyMap() : bundle;
+  }
+
+  /**
+   * Records one shard's lookup outcome. Counts records rather than distinct keys, so
+   * {@code hits + misses == records_looked_up} holds when a batch repeats a key. Membership is tested
+   * against the found set, bounded by the hit count, not the asked-about set, bounded by shard size.
+   *
+   * @param keysLookedUp every record key routed to this shard
+   * @param foundKeys    the subset present in the index
+   * @param elapsedMs    wall-clock spent reading this shard
+   */
+  public static void recordShardLookup(String caller, Collection<String> keysLookedUp,
+                                       Collection<String> foundKeys, long elapsedMs) {
+    if (keysLookedUp.isEmpty()) {
+      return;
+    }
+    Registry registry = Registry.getRegistry(RecordIndexMetricNames.REGISTRY_NAME);

Review Comment:
   Filed as #19759 with the deferred items listed; resolving on that basis.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5364228458

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 30dbc00d04a21cd671b1e02b527f3b340f828008 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431) 
   * 2c37a62e311d57eb3636dc5f28af1739788c98f9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432) 
   * a061dbc0124d8ebc62defdf41d2d428842a6b16c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3762969332


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/RecordIndexMetricNames.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Counter names for the record level index lookup phase, and the commit-boundary drain that publishes
+ * them.
+ *
+ * <p>Lives in hudi-client-common rather than hudi-spark-client so the shared commit path can reach it.
+ * Because the drain happens on that shared path, it applies to every engine write path -- Spark
+ * DataSource, Spark SQL and DeltaStreamer -- without any of them knowing about it.
+ */
+public class RecordIndexMetricNames {
+
+  /** Registry name, keyed per table in {@code Registry.REGISTRY_MAP}. */
+  public static final String REGISTRY_NAME = "HoodieRecordIndexLookup";
+
+  /** Prefix under which counters are stamped into commit metadata. */
+  public static final String COMMIT_METADATA_PREFIX = "hoodie.rli.lookup.";
+
+  /** Metric-name components used when the same counters are published to a reporter. */
+  public static final String METRIC_ACTION = "rli";
+  public static final String METRIC_QUALIFIER = "lookup";
+
+  // Callers. Counters are tagged so dedupe traffic is distinguishable from tag-location traffic.
+  public static final String CALLER_TAG_LOCATION = "tag";
+  public static final String CALLER_DEDUPE = "dedupe";
+
+  public static final String RECORDS_LOOKED_UP = "records_looked_up";
+  public static final String HITS = "hits";
+  public static final String MISSES = "misses";
+  public static final String SHARDS_READ = "shards_read";
+
+  private RecordIndexMetricNames() {
+  }
+
+  /** Counter key for a metric attributed to a caller, e.g. {@code tag.hits}. */
+  public static String key(String caller, String metric) {
+    return caller + "." + metric;
+  }
+
+  /**
+   * Snapshots this table's record index lookup counters into the commit's extra metadata, without
+   * consuming them.
+   *
+   * <p>Split from {@link #publishAndRelease} so the counters survive a commit that never lands: a
+   * conflict in preCommit or a failure in commit would otherwise destroy them and publish reporter
+   * gauges for work that was rolled back. The caller releases them only once the commit has succeeded.
+   *
+   * @param commitMetadata the extra-metadata map being assembled for this commit
+   * @param config         the write config, for the table name and the feature gate
+   * @return the snapshotted counters, to be handed to {@link #publishAndRelease} on success; empty
+   *         when the feature is off or nothing was recorded

Review Comment:
   🤖 nit: `snapshotIntoCommitMetadata` mutates `commitMetadata` in-place AND returns a separate `Map<String, Long>` — two output channels from one call. Have you considered returning a small result object (or a `Pair`) that holds both the snapshot and whatever the caller needs to pass to `publishAndRelease`, so the side-effectful mutation is not implicit in the signature?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/RecordIndexMetricNames.java:
##########
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.Collections;
+import java.util.Map;
+
+/**
+ * Counter names for the record level index lookup phase, and the commit-boundary drain that publishes
+ * them.
+ *
+ * <p>Lives in hudi-client-common rather than hudi-spark-client so the shared commit path can reach it.
+ * Because the drain happens on that shared path, it applies to every engine write path -- Spark
+ * DataSource, Spark SQL and DeltaStreamer -- without any of them knowing about it.

Review Comment:
   🤖 nit: the name `RecordIndexMetricNames` signals a constants class, but `snapshotIntoCommitMetadata` and `publishAndRelease` are substantive operations. Could you consider something like `RecordIndexLookupMetricsHelper` or `RecordIndexCommitDrain` — or splitting the constants into a separate `RecordIndexMetricNames` interface and putting the drain logic in a class with a name that reflects its responsibility?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): make record index lookups observable via executor metrics [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3817095063


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -142,9 +146,25 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
     return Option.empty();
   }
 
+  /**
+   * Carries the task thread's state onto the pool threads Hudi hands record writing to.
+   *
+   * <p>The metrics binding travels alongside the {@link TaskContext} for the same reason the
+   * TaskContext does: both are thread-local, and {@code BoundedInMemoryExecutor} and
+   * {@code DisruptorExecutor} run on threads the task did not create. Without this, a metric raised
+   * while writing records resolves against an unbound thread and is discarded.
+   *
+   * <p>Nothing unbinds afterwards. These pools are created and shut down per operation, so the thread
+   * does not outlive the binding; a pool that were shared across tables would need an unbind hook on
+   * the other side, which no current executor provides.
+   */
   @Override
   public Runnable getPreExecuteRunnable() {
     final TaskContext taskContext = TaskContext.get();
-    return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+    final Map<String, Registry> metricsBinding = ExecutorMetricsContext.capture();
+    return () -> {
+      TaskContext$.MODULE$.setTaskContext(taskContext);
+      ExecutorMetricsContext.bind(metricsBinding);

Review Comment:
   🤖 Since this override lives on the base `HoodieSparkTable`, every Spark write's executor pool threads (merge/insert/clustering/bootstrap all go through `getPreExecuteRunnable`) now get `bind(...)` and so report `isBound() == true`. That flips `Registry.getRegistry(name)` on those threads: a name miss now returns `NoOpRegistry` (discarded) instead of a process-wide `LocalRegistry` (collected). Have you confirmed nothing on those pool threads emits via `Registry.getRegistry`, so no unrelated metric gets silently dropped? Relatedly, during the write phase the task thread isn't inside a lookup `call()`'s bound scope, so `capture()` here looks like it returns an empty map — is the propagation actually carrying the table's bundle as the javadoc describes, or is it currently a no-op?
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5373812826

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8a71f71978edc6f83d37711a582e516b743aa7e7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858618050


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {

Review Comment:
   Not sure i understand this comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3853268963


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -255,6 +256,12 @@ public boolean commitStats(String instantTime, TableWriteStats tableWriteStats,
       return true;
     }
     extraMetadata = updateExtraMetadata(extraMetadata);
+    // The commit is the boundary for the record index lookup counters. Snapshot them into this commit's
+    // metadata now, but only release them from the registry once the commit has actually landed --
+    // hooked here rather than in updateExtraMetadata because that is shared with table-service
+    // scheduling, which would otherwise consume the counters into a compaction or clustering plan.
+    ExecutorMetrics.DrainedCounters executorCounters =
+        ExecutorMetrics.snapshotIntoCommitMetadata(extraMetadata.get(), config);

Review Comment:
   `snapshotIntoCommitMetadata` mutates the enriched `extraMetadata` map, and the same `Option` goes to `runTableServicesInline` at L302. From there `scheduleTableServiceInternal` re-enriches with `putAll`, `ScheduleCompactionActionExecutor:109` puts it on the plan, and `CompactHelpers:85` copies the plan's extra metadata into the completed compaction commit, so an inline compaction carries the triggering commit's `hoodie.rli.lookup.*` values. That is the misattribution the comment above says this placement avoids, and no new test enables inline services. Could we stamp into a copy used only for `CommitUtils.buildMetadata`, and add one MOR inline-compaction case asserting the compaction commit has no `hoodie.rli.lookup.*` key?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {

Review Comment:
   A commit that performs no lookup (insert-only, bulk_insert, an empty micro-batch) takes this `continue`, so the `rli.lookup.*` gauges registered by an earlier commit keep that commit's values and a scheduled reporter (pushgateway) re-emits them indefinitely; per key, `dedupe.*` goes stale while `tag.*` updates, so the two sinks disagree. HUDI-3373 (#4760) added `HoodieMetrics.updateMetricsForEmptyData` for exactly this on the `commit.*` gauges. Could we publish explicit zeros for the group's counter names when the registry is missing or all-zero (keeping the timeline skip), or add these names to `updateMetricsForEmptyData`?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {
+        continue;
+      }
+      counts.forEach((name, value) ->
+          commitMetadata.put(metricRegistry.commitMetadataPrefix() + name, String.valueOf(value)));
+      drained.add(new Drained(metricRegistry, registry, counts));
+    }
+    return drained.isEmpty() ? DrainedCounters.EMPTY : new DrainedCounters(drained);
+  }
+
+  /**
+   * Release subtracts what was published rather than clearing, so a straggler task's update arriving after
+   * the snapshot survives. Publishing here rather than letting the reporter scrape is what lets both sinks
+   * work at once: {@link Registry#getAllMetrics} consumes the registry when it scrapes.
+   */
+  public static void publishAndRelease(DrainedCounters counters, HoodieMetrics hoodieMetrics) {

Review Comment:
   A failed or abandoned attempt's counters are stamped on whatever commits next, so that commit's `key_count` no longer describes its own records. Your earlier branch fixed exactly this at `dd4a48c` (clear at `startCommit`: "an abandoned 80-key attempt followed by a 10-key commit reported 80"); this PR reverses it and asserts the opposite. Could we restore clear-at-`startCommit`, flip the `AcrossFailedCommit` assertion to "the retry reports only its own lookups", and add an insert-after-failed-upsert case?
   
   <details><summary>Why clear-at-startCommit is safe, and what else picks up leftovers</summary>
   
   - `startCommit` precedes every in-tree lookup: `HoodieSparkSqlWriter:533` before `:553 handleInsertDuplicates`; `StreamSync:877` before `:1104`; write-path `tagLocation` runs inside `upsert/delete`.
   - Leftovers also land on `commitTableChange` (ALTER SCHEMA, `BaseHoodieWriteClient:1872`) and on a successful empty commit (`hoodie.allow.empty.commit=false` returns at L255, above the snapshot).
   - With `hoodie.metrics.on=true` the carry-over does not hold anyway: `DefaultSource:194 cleanup()` -> `Metrics.shutdownAllMetrics()` -> `Registry.getAllMetrics(true, true)` clears every registry after each DataSource write.
   </details>



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDReadClient.java:
##########
@@ -210,8 +212,16 @@ public JavaRDD<HoodieRecord<T>> filterExists(JavaRDD<HoodieRecord<T>> hoodieReco
    * @return Tagged RDD of Hoodie records
    */
   public JavaRDD<HoodieRecord<T>> tagLocation(JavaRDD<HoodieRecord<T>> hoodieRecords) throws HoodieIndexException {
-    return HoodieJavaRDD.getJavaRDD(
-        index.tagLocation(HoodieJavaRDD.of(hoodieRecords), context, hoodieTable));
+    // Lookups driven from the read client are dedupe traffic, not tag-location traffic. Label them so
+    // the two are distinguishable in the reported counters. Driver-side only: the label is
+    // captured when the lookup closure is built.
+    String previousCaller = RecordIndexLookupMetrics.setCaller(RecordIndexMetricNames.CALLER_DEDUPE);

Review Comment:
   `CALLER_DEDUPE` has no test coverage: `grep -rn CALLER_DEDUPE` under `src/test` returns nothing, and `TestRliMetricsOnStreamerPath` asserts only `tag.*`. It is reachable on the DataSource path too (`insert.drop.duplicates` -> `HoodieSparkSqlWriter:553` -> `resolveDuplicates`), which is cheaper to test than a streamer arm. Also `checkExists` (L188) calls `index.tagLocation` directly, so it is labelled `tag` while `filterExists` is `dedupe`, contrary to the comment above. Could we add one DataSource case with `insert.drop.duplicates=true` asserting both `dedupe.*` and `tag.*` on the same commit, and set the label once for the whole read client?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = context.getMetricRegistry(config.getTableName(),

Review Comment:
   `getMetricRegistry` starts with `tableName.isEmpty()`, and `config.getTableName()` can be null: `TBL_NAME` has no default and `Builder.validate()` only requires `BASE_PATH`. Pre-PR that line sat behind `hoodie.metrics.on`; now every RLI `tagLocation` reaches it, so the public `SparkRDDReadClient(ctx, basePath, sqlCtx, RECORD_INDEX)` ctor (L104) or any write config built without `forTable()` NPEs on an RLI table. Could we skip collection when the table name is null or empty?
   ```suggestion
         String tableName = config.getTableName();
         if (tableName == null || tableName.isEmpty()) {
           continue;
         }
         Registry registry = context.getMetricRegistry(tableName,
   ```



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",

Review Comment:
   Once written these keys are permanent timeline content, so the naming is worth settling now. No existing extra-metadata key starts with `hoodie.` (`schema`, `deltastreamer.checkpoint.key`, `latest_schema`, and #18183's `hudi.version` / `engine` / `config.<key>`, which deliberately prefixes real config keys with `config.`), so `hoodie.` reads as a config key that does not exist. The full key also repeats itself: `hoodie.rli.lookup.tag.lookup_record_index_key_count`. Could the prefix be `rli.lookup.` (or `metrics.record.index.lookup.`) and the counters drop the `lookup_record_index_` segment, e.g. `rli.lookup.tag.key_count`?



##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestRliMetricsOnStreamerPath.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities.deltastreamer;
+
+import org.apache.hudi.common.config.HoodieMetadataConfig;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.WriteOperationType;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.testutils.HoodieTestUtils;
+import org.apache.hudi.config.HoodieIndexConfig;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import org.junit.jupiter.api.Tag;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The RLI lookup counters must reach commit metadata on the DeltaStreamer path, not only on the Spark DataSource path.
+ */
+@Tag("functional")
+public class TestRliMetricsOnStreamerPath extends HoodieDeltaStreamerTestBase {
+
+  /** Selects the global or partitioned record index. */
+  private static void enableRecordIndex(HoodieDeltaStreamer.Config cfg, boolean partitioned) {
+    cfg.configs.add(HoodieMetadataConfig.ENABLE.key() + "=true");
+    cfg.configs.add(HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key() + "=" + !partitioned);
+    cfg.configs.add(HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key() + "=" + partitioned);
+    cfg.configs.add(HoodieIndexConfig.INDEX_TYPE.key() + "="
+        + (partitioned ? "RECORD_LEVEL_INDEX" : "GLOBAL_RECORD_LEVEL_INDEX"));
+  }
+
+  private static Map<String, String> rliCountersOnLatestCommit(String tableBasePath) throws Exception {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HoodieTestUtils.getDefaultStorageConf())
+        .setBasePath(tableBasePath)
+        .build();
+    metaClient.reloadActiveTimeline();
+    HoodieInstant lastInstant = metaClient.getActiveTimeline()
+        .getCommitsTimeline().filterCompletedInstants().lastInstant().get();
+    HoodieCommitMetadata commitMetadata = metaClient.getActiveTimeline().readCommitMetadata(lastInstant);
+    Map<String, String> rli = new HashMap<>();
+    commitMetadata.getExtraMetadata().forEach((k, v) -> {
+      if (k.startsWith(RecordIndexMetricNames.COMMIT_METADATA_PREFIX)) {
+        rli.put(k, v);
+      }
+    });
+    return rli;
+  }
+
+  private static String tagKey(String metric) {
+    return RecordIndexMetricNames.COMMIT_METADATA_PREFIX
+        + RecordIndexMetricNames.key(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric);
+  }
+
+  @ParameterizedTest
+  @ValueSource(booleans = {false, true})
+  public void testRliCountersReachCommitMetadataOnStreamerPath(boolean partitioned) throws Exception {
+    String label = partitioned ? "partitioned" : "global";
+    String tableBasePath = basePath + "/test_rli_metrics_streamer_" + label;
+
+    // Sync 1 -- build the table and the record index.
+    HoodieDeltaStreamer.Config insertCfg =
+        TestHoodieDeltaStreamer.TestHelpers.makeConfig(tableBasePath, WriteOperationType.INSERT);
+    enableRecordIndex(insertCfg, partitioned);
+    new HoodieDeltaStreamer(insertCfg, jsc).sync();
+
+    // Sync 2 -- upsert, which tags incoming keys against the record index.
+    HoodieDeltaStreamer.Config upsertCfg =
+        TestHoodieDeltaStreamer.TestHelpers.makeConfig(tableBasePath, WriteOperationType.UPSERT);
+    enableRecordIndex(upsertCfg, partitioned);
+    new HoodieDeltaStreamer(upsertCfg, jsc).sync();
+
+    Map<String, String> counters = rliCountersOnLatestCommit(tableBasePath);
+
+    System.out.println("\n===== DeltaStreamer (" + label + " RLI) -- RLI counters on the commit =====");
+    if (counters.isEmpty()) {
+      System.out.println("  (none found)");
+    } else {
+      counters.entrySet().stream()
+          .sorted(Map.Entry.comparingByKey())
+          .forEach(e -> System.out.println(String.format("  %-52s %s", e.getKey(), e.getValue())));
+    }
+    System.out.println("==========================================================\n");
+
+    assertFalse(counters.isEmpty(),
+        "the commit-boundary drain must fire on the DeltaStreamer path; hudi-utilities never calls "
+            + "Metrics.shutdownAllMetrics, so nothing else would publish these");
+
+    String lookedUp = tagKey(RecordIndexMetricNames.KEY_COUNT);
+    assertTrue(counters.containsKey(lookedUp),
+        "tag-location traffic must be attributed on the streamer path too; got " + counters.keySet());
+
+    long records = Long.parseLong(counters.get(lookedUp));
+    long hits = Long.parseLong(counters.get(tagKey(RecordIndexMetricNames.KEY_HIT_COUNT)));
+    long misses = Long.parseLong(counters.get(tagKey(RecordIndexMetricNames.KEY_MISS_COUNT)));
+    assertTrue(records > 0, "the upsert sync looked up at least one key");
+    assertEquals(records, hits + misses, "hits + misses must account for every key looked up");

Review Comment:
   `records > 0` plus `records == hits + misses` cannot fail on a doubled count: `misses` is computed as `records - hits` at the emission site (`RecordIndexLookupMetrics:111`), so the identity holds by construction. This is the only DeltaStreamer-path test and the values are deterministic (CI logged `1000 / 500 / 500` on both arms: `sourceLimit=1000`, then 500 updates + 500 inserts). Could we assert the exact counts instead?
   ```suggestion
       assertEquals(1000L, records, "the upsert sync looked up every key from the first sync");
       assertEquals(500L, hits, "the 500 updates hit the index");
       assertEquals(500L, misses, "the 500 fresh inserts missed");
   ```



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {

Review Comment:
   This gate makes the RLI registry the first thing to enter `DISTRIBUTED_REGISTRY_MAP` without opt-in (pre-PR: only `DistributedRegistryUtil`, behind `hoodie.metrics.on` AND `hoodie.metrics.executor.enable`). That static map is captured into every `HoodieSparkEngineContext.map/mapToPair/flatMap` closure (L141-206), so every such job for every table in the JVM now carries one accumulator per RLI table per task, and nothing evicts. Could the gate follow `LOCK_METRICS_ENABLE` (`.withInferFunction(cfg -> cfg.getBoolean(TURN_METRICS_ON))`), and could this registry stay out of the engine-context captures, which only need the FS registries?
   
   <details><summary>Per-task cost</summary>
   
   The closure carries an empty `copyAndReset` copy (cheap), but per task Spark registers it with the `TaskContext`, ships an update back even when zero, and retains one unnamed `AccumulableInfo` per task per accumulator in the driver AppStatusStore/UI and the event log. The single-table overhead run in the description cannot show this.
   </details>



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/DistributedRegistry.java:
##########
@@ -68,9 +79,36 @@ public void add(String name, long value) {
 
   @Override
   public void set(String name, long value) {
+    // Last-writer-wins is neither commutative nor associative, and the driver merges executor copies in
+    // an unspecified order. Driver only; executors use increment()/add().
+    if (TaskContext.get() != null) {

Review Comment:
   These guards throw from inside a task, and `release()`'s copy runs at `BaseHoodieWriteClient:287` after the commit has landed, inside a `try` that catches only `IOException`, so a metrics problem would surface as a failed write on a completed instant. That contradicts the policy stated in `NoOpRegistry` ("a metrics gap must not fail a write"), and `Metrics.registerGauge` already swallows for the same reason. Could both guards `LOG.warn` and return instead, and the drain at L264/L287 be wrapped in a catch-all?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/RecordIndexMetricNames.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+/**
+ * Counter names for the record index lookup phase. Collection itself is generic -- see
+ * {@link ExecutorMetricRegistry} and {@link ExecutorMetrics}.
+ */
+public class RecordIndexMetricNames {
+
+  /** Scoping, prefix and reporter naming all live on the enum entry. */
+  public static final String REGISTRY_NAME = ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.registryName();
+
+  public static final String COMMIT_METADATA_PREFIX =
+      ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.commitMetadataPrefix();
+
+  // Counters are tagged by caller so dedupe traffic is distinguishable from tag-location traffic.
+  public static final String CALLER_TAG_LOCATION = "tag";
+  public static final String CALLER_DEDUPE = "dedupe";
+
+  public static final String KEY_COUNT = "lookup_record_index_key_count";

Review Comment:
   Two small things on the names. (1) The value counts records, not distinct keys (`keysToLookup` has one entry per `HoodieRecord`), while the name says `key_count` and the config doc says "records looked up". (2) L37-43 re-declare the literal strings of `HoodieMetadataMetrics.LOOKUP_RECORD_INDEX_*`, dead since `2a0d2aeef6a0` (HUDI-7391), and `HoodieBackedTableMetadata:356` still carries the `TODO [HUDI-9544]` the description says this closes. Could the counter be `record_count` (or the doc say keys), and could the dead constants be referenced or deleted and the TODO updated?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/RecordIndexMetricNames.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+/**
+ * Counter names for the record index lookup phase. Collection itself is generic -- see
+ * {@link ExecutorMetricRegistry} and {@link ExecutorMetrics}.
+ */
+public class RecordIndexMetricNames {
+
+  /** Scoping, prefix and reporter naming all live on the enum entry. */
+  public static final String REGISTRY_NAME = ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.registryName();
+
+  public static final String COMMIT_METADATA_PREFIX =
+      ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.commitMetadataPrefix();
+
+  // Counters are tagged by caller so dedupe traffic is distinguishable from tag-location traffic.
+  public static final String CALLER_TAG_LOCATION = "tag";
+  public static final String CALLER_DEDUPE = "dedupe";
+
+  public static final String KEY_COUNT = "lookup_record_index_key_count";
+  public static final String KEY_HIT_COUNT = "lookup_record_index_key_hit_count";
+  public static final String KEY_MISS_COUNT = "lookup_record_index_key_miss_count";
+  public static final String SHARDS_READ = "lookup_record_index_shards_read";
+  /** Wall-clock spent in the shard read, summed across shards. Revives the third dead upstream constant,
+   * {@code HoodieMetadataMetrics.LOOKUP_RECORD_INDEX_TIME_STR}. */
+  public static final String LOOKUP_TIME = "lookup_record_index_time";

Review Comment:
   `SparkRDDWriteClient:189` already publishes `index.lookup.duration` via `HoodieMetrics.updateIndexMetrics` (driver wall-clock for the whole `tagLocation`), so this adds a second lookup-time metric with a different meaning (sum of per-shard executor time). Could we drop `LOOKUP_TIME`, or state in the javadoc and the config doc how the two differ?



##########
hudi-common/src/main/java/org/apache/hudi/common/config/metrics/HoodieMetricsConfig.java:
##########
@@ -109,6 +109,17 @@ public class HoodieMetricsConfig extends HoodieConfig {
       .sinceVersion("0.13.0")
       .withDocumentation("Enable metrics for locking infra. Useful when operating in multiwriter mode");
 
+  public static final ConfigProperty<Boolean> RLI_LOOKUP_METRICS_ENABLE = ConfigProperty

Review Comment:
   nit, feel free to ignore: every sibling boolean here has a `Builder` setter (`withExecutorMetrics`, `withLockingMetrics`, `withCompactionLogBlockMetrics`). Could this one get `withRecordIndexLookupMetrics(boolean)` for parity?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {
+    DISTRIBUTED_REGISTRY_MAP.remove(tableName.isEmpty() ? registryName : tableName + "." + registryName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {

Review Comment:
   The process-wide map is what forces the SHA-256 digest, the stale-context branch, the replacement branch below and `removeMetricRegistryForTesting`, and it is the source of the null-`tableName` NPE and the unbounded growth raised in `RecordIndexLookupMetrics`. #19063 lists "tying registry lifetime to `HoodieEngineContext` / write client rather than the JVM process" as the intended follow-up, and it looks reachable: `StreamSync:1104` reuses the write client's context for dedupe, and `HoodieSparkSqlWriter:553` has `client.getEngineContext` in scope (`DataSourceUtils:298` currently builds a throwaway one). Could the registry be an instance field on `HoodieSparkEngineContext` keyed by base path, with `commitStats` reading it off `context`?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/DistributedRegistry.java:
##########
@@ -48,9 +51,17 @@ public String getName() {
   public void register(JavaSparkContext jsc) {
     if (!isRegistered()) {
       jsc.sc().register(this);
+      // Only when this call actually registers: stamping unconditionally would re-brand an accumulator
+      // bound to a dead context and mask the staleness this field exists to detect.
+      this.registeredAppId = jsc.sc().applicationId();
     }
   }
 
+  /** False when bound to a different (typically stopped) context, meaning it must be recreated. */
+  public boolean isRegisteredWith(JavaSparkContext jsc) {

Review Comment:
   `isRegisteredWith` + the `compute` rewrite, the `set()` executor guard and release-at-commit are gaps 2, 7 and 8 of #19063 (the guard is proposed there verbatim), and they fix `hoodie.metrics.executor.enable` on master today for the existing `HoodieWrapperFileSystem` user. That issue's own breakdown puts them in PR 1 / PR 2, and the description here does not reference it. Could the `DistributedRegistry` / `getMetricRegistry` hardening go out as its own PR under #19063, leaving this one to add the counters on top?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la
+ */
+abstract class RliLookupMetricsTestBase extends RecordLevelIndexTestBase {

Review Comment:
   The drain has one engine-agnostic hook (`commitStats:264/287`), but the functional suite re-covers it 30 times: 79 full MDT+RLI table writes and 30 SparkContext restarts on the single UT_FT_2 job. Could we consolidate to roughly 4 classes / 28 writes?
   
   - drop `TestRliLookupMetricsOnDataSourceMor*`: no instrumented code branches on table type (22 writes)
   - fold the one distinct SQL case (non-prepped UPDATE exact counts) into the existing `TestGlobalRecordLevelIndexWithSQL`
   - replace `TestRliLookupMetricsReporting` (8 writes, 4 SparkContexts) with one `TestHoodieMetrics` unit test calling `publishAndRelease`
   - drop `TestRliLookupMetricsMultiTable`: its tables have different `TBL_NAME`s, which separate the keys on their own, so it cannot exercise the digest (`TestRecordIndexMetricNames.countersAreScopedByBasePathNotOnlyByTableName` does)
   - drop the AcrossFailedCommit Partitioned twin and the streamer partitioned arm (covered by `TestRliLookupMetricsOnDataSourcePartitioned`)
   - unit side: merge `TestExecutorMetricsGenericity` into `TestRecordIndexMetricNames`; fold `TestRegistryExecutorLookup` into the existing `TestRegistry` (3 of 7 pass on master unchanged); merge the two self-managed-SparkContext `DistributedRegistry` files, dropping the 16-thread test (`ConcurrentHashMap.compute` serialises per key, so it cannot fail) and `testRepeatedEvaluationDoubleCounts` (pins Spark, not Hudi)



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRliLookupMetricsOnSparkSql.scala:
##########
@@ -0,0 +1,134 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import org.apache.spark.sql.SaveMode
+import org.junit.jupiter.api.{Tag, Test}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue}
+
+/** Record level index lookup counters on the Spark SQL write path. */
+@Tag("functional")
+class TestRliLookupMetricsOnSparkSql extends RliLookupMetricsTestBase {
+
+  private val sqlTable = "rli_lookup_metrics_tbl"
+  private val numSeedRecords = 60
+
+  /**
+   * Seeds a table through the DataSource so the record index exists, then exposes it to SQL and applies
+   * the index settings as session configs -- index type is a write config, not a table property.
+   */
+  private def seedTableAndRegisterForSql(): Unit = {
+    doWriteAndValidateDataAndRecordIndex(rliOpts, INSERT_OPERATION_OPT_VAL, SaveMode.Overwrite,
+      validate = false, numInserts = numSeedRecords)
+
+    spark.sql(s"drop table if exists $sqlTable")
+    spark.sql(s"create table $sqlTable using hudi location '$basePath'")
+
+    spark.sql("set hoodie.write.lock.provider = org.apache.hudi.client.transaction.lock.InProcessLockProvider")
+    spark.sql(s"set ${HoodieMetadataConfig.ENABLE.key} = true")
+    spark.sql(s"set ${HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key} = ${!isPartitionedRli}")
+    spark.sql(s"set ${HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key} = $isPartitionedRli")
+    spark.sql(s"set ${HoodieIndexConfig.INDEX_TYPE.key} = " +
+      (if (isPartitionedRli) "RECORD_LEVEL_INDEX" else "GLOBAL_RECORD_LEVEL_INDEX"))
+
+    clearRliRegistry()
+  }
+
+  /**
+   * The default path. Optimized writes make UPDATE a prepped write, so no index lookup happens and no
+   * counters are produced. Documented behaviour, asserted so it cannot change unnoticed.
+   */
+  @Test
+  def testUpdateWithOptimizedWritesPerformsNoLookup(): Unit = {
+    seedTableAndRegisterForSql()
+    spark.sql(s"set ${SPARK_SQL_OPTIMIZED_WRITES.key} = true")
+
+    spark.sql(s"update $sqlTable set rider = 'rider-optimized'")
+
+    val counters = rliCountersFromLatestCommit()
+    report(s"Spark SQL UPDATE, optimized writes ON ($indexLabel) -- expected empty", counters)
+    assertTrue(counters.isEmpty,

Review Comment:
   This assertion has no positive control in the method: the seed is an INSERT (no lookup), so `counters.isEmpty` also holds if the feature is broken or mis-gated. Could the test first assert that a preceding upsert did publish (as `TestRliLookupMetricsOnDataSource:85` does) before asserting the prepped UPDATE is silent?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la

Review Comment:
   nit: the sentence is cut off.
   ```suggestion
    * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the latest completed commit.
   ```



##########
hudi-common/src/test/java/org/apache/hudi/common/TestRegistryExecutorLookup.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common;
+
+import org.apache.hudi.common.metrics.ExecutorMetricsContext;
+import org.apache.hudi.common.metrics.LocalRegistry;
+import org.apache.hudi.common.metrics.Registry;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotSame;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/** Characterizes how an executor reaches a driver-published registry by name. */
+public class TestRegistryExecutorLookup {

Review Comment:
   Same as the genericity test: `publishAsDriverDoes` and the bare-name lookups leave non-zero entries in the process-wide `Registry.REGISTRY_MAP` with no `@AfterEach`; `TestRegistry.testGetAllMetrics:64` in this package only survives because its own `flush=true` scrape wipes them. Could this file get the same cleanup (or be folded into `TestRegistry`)?



##########
hudi-common/src/main/java/org/apache/hudi/common/config/metrics/HoodieMetricsConfig.java:
##########
@@ -109,6 +109,17 @@ public class HoodieMetricsConfig extends HoodieConfig {
       .sinceVersion("0.13.0")
       .withDocumentation("Enable metrics for locking infra. Useful when operating in multiwriter mode");
 
+  public static final ConfigProperty<Boolean> RLI_LOOKUP_METRICS_ENABLE = ConfigProperty
+      .key(METRIC_PREFIX + ".rli.lookup.enable")
+      .defaultValue(true)
+      .markAdvanced()
+      .sinceVersion("1.3.0")
+      .withDocumentation("Collect counters for the record level index lookup phase (records looked up, "

Review Comment:
   The doc reads as engine-neutral, but only the Spark RLI lookup emits: Flink maps `RECORD_LEVEL_INDEX` to `FlinkInMemoryStateIndex` and the Java client has no RLI case, so on those engines the drain finds no registry. Two write clients on the same table in one JVM (OCC, in-process async services) also share one registry and both snapshot before either releases. Could the doc say "Spark only" and note that counters are per table per JVM rather than per writer?



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metrics/TestDistributedRegistry.java:
##########
@@ -184,6 +187,131 @@ public void testAddMetricsParallel() {
     Assertions.assertEquals(finalExpectedSum, metricCounts.get(METRIC_1));
   }
 
+  @Test
+  public void testSetThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testSetOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: set() invoked on an executor must fail - it is non-commutative under accumulator merges.
+    // The UnsupportedOperationException thrown on the executor surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.set() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.set(METRIC_1, value);
+          return null;
+        }, 1));
+  }
+
+  @Test
+  public void testReleaseThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testReleaseOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: release() invoked on an executor must fail - clamping and eviction are order-dependent
+    // under accumulator merges. The UnsupportedOperationException surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.release() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.release(Collections.singletonMap(METRIC_1, (long) value));
+          return null;
+        }, 1));
+  }
+
+  /**
+   * Asserts the job failed because the executor-side guard fired, not for some unrelated reason such as a serialization error.
+   */
+  private static void assertFailsOnExecutorWith(String expectedMessage, Executable executable) {

Review Comment:
   nit, feel free to ignore: matching the exception message substring means a reword of the guard text breaks these two tests without any behaviour change. Could `assertFailsOnExecutorWith` match `UnsupportedOperationException` on the cause chain instead?



##########
hudi-client/hudi-client-common/src/test/java/org/apache/hudi/metrics/TestExecutorMetricsGenericity.java:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.ExecutorMetricsContext;
+import org.apache.hudi.common.metrics.LocalRegistry;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import org.junit.jupiter.api.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * The claim this feature exists to support is that adding a class of executor metric costs a declaration
+ * plus two lines at the emission site. These tests hold that claim to a measurement rather than an
+ * argument, by collecting a group the shipping code has never heard of.
+ */
+public class TestExecutorMetricsGenericity {
+
+  private static final String BASE_PATH = "file:///tmp/test_generic_metrics";
+
+  /** A class of metric added by a hypothetical future contributor: a name, a prefix, and no config. */
+  private static final ExecutorMetricGroup STORAGE_CALLS = new ExecutorMetricGroup() {
+    @Override
+    public String registryName() {
+      return "HoodieStorageCalls";
+    }
+
+    @Override
+    public String commitMetadataPrefix() {
+      return "hoodie.storage.calls.";
+    }
+
+    @Override
+    public String metricAction() {
+      return "storage";
+    }
+
+    @Override
+    public String metricQualifier() {
+      return "calls";
+    }
+
+    @Override
+    public boolean isEnabled(HoodieWriteConfig config) {
+      return true;
+    }
+
+    @Override
+    public String scopedName(String basePath) {
+      return registryName() + ".test";
+    }
+  };
+
+  private static HoodieWriteConfig config() {
+    return HoodieWriteConfig.newBuilder().withPath(BASE_PATH).forTable("generic_metrics_table").build();
+  }
+
+  private static Registry seed(HoodieWriteConfig cfg) {
+    Registry registry = new LocalRegistry(STORAGE_CALLS.scopedName(cfg.getBasePath()));
+    Registry.REGISTRY_MAP.put(

Review Comment:
   These `put`s go into the process-wide `Registry.REGISTRY_MAP` with no `@AfterEach`, and `twoGroupsInOneCommitDoNotMix` never releases, so `open=3` / `tag.lookup_record_index_key_count=11` outlive the class under `reuseForks=true`; the sibling `TestRecordIndexMetricNames:52` cleans up. Also the `allMatch` at L117 cannot fail (the map was just filled from a single-group list). Could this get the same `@AfterEach removeIf`, and the `allMatch` be dropped?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la
+ */
+abstract class RliLookupMetricsTestBase extends RecordLevelIndexTestBase {
+
+  /** Overridden by the partitioned subclasses; both variants are separate closures on separate paths. */
+  protected def isPartitionedRli: Boolean = false
+
+  /**
+   * Table type under test. Tagging is an index-level concern and does not branch on table type, so MOR
+   * is expected to behave identically -- the MOR subclasses exist to prove that rather than assume it.
+   */
+  protected def tableTypeOpt: String = DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL
+
+  protected def indexLabel: String = {
+    val idx = if (isPartitionedRli) "partitioned RLI" else "global RLI"
+    val tt = if (tableTypeOpt == DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL) "MOR" else "COW"
+    s"$idx, $tt"
+  }
+
+  /**
+   * `commonOpts` turns the global record index on, so the metadata-partition flags and the index type
+   * have to be flipped together to select the partitioned variant.
+   */
+  protected def rliOpts: Map[String, String] = {
+    val withTableType = Map(DataSourceWriteOptions.TABLE_TYPE.key -> tableTypeOpt)
+    if (isPartitionedRli) {
+      commonOpts ++ withTableType ++ Map(
+        HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false",
+        HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true",
+        HoodieIndexConfig.INDEX_TYPE.key -> "RECORD_LEVEL_INDEX")
+    } else {
+      commonOpts ++ withTableType ++ Map(
+        HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true",
+        HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false",
+        HoodieIndexConfig.INDEX_TYPE.key -> "GLOBAL_RECORD_LEVEL_INDEX")
+    }
+  }
+
+  protected def counterKey(caller: String, metric: String): String =
+    RecordIndexMetricNames.COMMIT_METADATA_PREFIX + RecordIndexMetricNames.key(caller, metric)
+
+  protected def tagKey(metric: String): String =
+    counterKey(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric)
+
+  /**
+   * A caller that looked something up stamps its full counter set, zeros included, so an absent key means
+   * that caller contributed nothing at all. The default is defensive against exactly that case.
+   */
+  protected def counterOrZero(counters: Map[String, String], caller: String, metric: String): Long =
+    counters.getOrElse(counterKey(caller, metric), "0").toLong
+
+  /** The counters as an operator would read them: off the latest completed commit. */
+  protected def rliCountersFromLatestCommit(): Map[String, String] = {
+    metaClient.reloadActiveTimeline()
+    val lastInstant = metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants().lastInstant().get()
+    metaClient.getActiveTimeline.readCommitMetadata(lastInstant).getExtraMetadata.asScala.toMap
+      .filter { case (k, _) => k.startsWith(RecordIndexMetricNames.COMMIT_METADATA_PREFIX) }
+  }
+
+  /** Leftover counters from a previous write would otherwise be folded into the next commit. */
+  protected def clearRliRegistry(): Unit = {

Review Comment:
   `clearRliRegistry()` is a no-op at all five call sites: `@TempDir` is per method, so the base-path digest (and the registry key) is unique per test, and the seeding write is an INSERT that this suite itself asserts produces no counters. Could the helper and its calls be deleted, so the tests do not imply a cross-write leak that the per-method temp dir already rules out?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = context.getMetricRegistry(config.getTableName(),
+          metricRegistry.scopedName(config.getBasePath()));
+      // Only the accumulator-backed registry aggregates back to the driver, so anything else is left out
+      // rather than bound: a bound LocalRegistry would collect on the executor and be dropped on the floor,
+      // whereas leaving it out makes the lookup resolve to a no-op that reports nothing.
+      if (registry instanceof DistributedRegistry) {
+        bundle.put(metricRegistry.registryName(), registry);
+      }
+    }
+    return bundle.isEmpty() ? Collections.emptyMap() : bundle;
+  }
+
+  /**
+   * Records one shard's lookup outcome. Counts records rather than distinct keys, so
+   * {@code hits + misses == records_looked_up} holds when a batch repeats a key. Membership is tested
+   * against the found set, bounded by the hit count, not the asked-about set, bounded by shard size.
+   *
+   * @param keysLookedUp every record key routed to this shard
+   * @param foundKeys    the subset present in the index
+   * @param elapsedMs    wall-clock spent reading this shard
+   */
+  public static void recordShardLookup(String caller, Collection<String> keysLookedUp,

Review Comment:
   The query read path (`PartitionedRecordLevelIndexSupport.scala:107`, 1-arg ctor, `caller = null`) and the disabled path both still start the timer, run the O(n) `found::contains` scan and build `"null.<metric>"` keys, all into `NoOpRegistry`. Could this return before the scan when the registry is a no-op (or, if the bundle is passed in directly, when it is null)?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",
+      "rli",
+      "lookup",
+      HoodieWriteConfig::isRecordIndexLookupMetricsEnabled);
+
+  private final String registryName;
+  private final String commitMetadataPrefix;
+  private final String metricAction;
+  private final String metricQualifier;
+  private final Predicate<HoodieWriteConfig> enabled;
+
+  ExecutorMetricRegistry(String registryName, String commitMetadataPrefix, String metricAction,
+                         String metricQualifier, Predicate<HoodieWriteConfig> enabled) {
+    this.registryName = registryName;
+    this.commitMetadataPrefix = commitMetadataPrefix;
+    this.metricAction = metricAction;
+    this.metricQualifier = metricQualifier;
+    this.enabled = enabled;
+  }
+
+  /** The bare name emitting code passes to {@link Registry#getRegistry(String)}. */
+  @Override
+  public String registryName() {
+    return registryName;
+  }
+
+  @Override
+  public String commitMetadataPrefix() {
+    return commitMetadataPrefix;
+  }
+
+  @Override
+  public String metricAction() {
+    return metricAction;
+  }
+
+  @Override
+  public String metricQualifier() {
+    return metricQualifier;
+  }
+
+  /** Gating here, rather than in the drain, is what lets a new class of metric need no new config. */
+  @Override
+  public boolean isEnabled(HoodieWriteConfig config) {
+    return enabled.test(config);
+  }
+
+  /**
+   * Driver-side {@code REGISTRY_MAP} key. Table name alone is not an identity: two tables can share one
+   * and would then share a registry. Executors use the bare {@link #registryName()}.
+   */
+  @Override
+  public String scopedName(String basePath) {

Review Comment:
   Because this name contains a `.`, `Registry.getAllMetrics(true, true)` (run by `Metrics.shutdown`, i.e. after every DataSource write with metrics on) skips the common prefix and republishes any leftover counters as `<table>.HoodieRecordIndexLookup.<12-hex>.tag.*`, a per-base-path metric name, while also clearing the registry. Could this registry be excluded from the common scrape, or given a digest-free name for it?



##########
hudi-io/src/main/java/org/apache/hudi/common/metrics/Registry.java:
##########
@@ -190,6 +199,28 @@ static void setRegistries(Collection<Registry> registries) {
    */
   void set(String name, long value);
 
+  /**
+   * Subtract a set of counts previously read out of this registry, clamping every counter at zero.
+   *
+   * Used to hand a batch of counters over to a consumer that owns them from then on -- the commit-boundary
+   * drain for the record index lookup counters -- without discarding whatever arrived after they were read.
+   *
+   * Clamping is what distinguishes this from {@code add(name, -value)}. The registry can be emptied
+   * underneath a caller by an unrelated destructive scrape ({@link #getAllMetrics(boolean, boolean)} with
+   * {@code flush=true} clears every registry in the process), and an unbounded subtraction would then leave
+   * negative counters behind for good.
+   *
+   * The default is a best-effort read-modify-write. Implementations able to do this atomically should
+   * override it, and should drop counters that reach zero rather than leaving them at zero, so a registry
+   * nobody is writing to reads as empty.
+   *
+   * @param counts the counts to release, as returned by {@link #getAllCounts(boolean)}.
+   */
+  default void release(Map<String, Long> counts) {

Review Comment:
   This default and the `LocalRegistry` override are unreachable in production: the only `release` caller is `ExecutorMetrics:94`, whose registry is always a `DistributedRegistry` (Flink/Java never create the drain key), and only `TestDistributedRegistry` exercises `release`. Could the default be dropped (leaving `release` on `DistributedRegistry`, or abstract), or a `LocalRegistry` test added if it is meant to be supported?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {

Review Comment:
   Two things here. (1) In local mode `setRegistries` also indexes the same registry under `Registry.makeKey("", prefixedName)`, so this remover and the staleness branch below evict only the `table::registry` key and the accumulator bound to the stopped context survives under `::table.registry`. (2) This is the only `*ForTesting` public static in `src/main`; the convention is `@VisibleForTesting` on a normally named method. Could it also drop the `""`-keyed entry and be renamed `removeMetricRegistry`?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricGroup.java:
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.config.HoodieWriteConfig;
+
+/**
+ * One class of executor-collected metric. Implemented by {@link ExecutorMetricRegistry}, which enumerates
+ * the ones that ship; kept an interface so the collection machinery can be exercised against a group it
+ * does not know about.
+ */
+public interface ExecutorMetricGroup {

Review Comment:
   This interface has one production implementer (the enum, one constant), both `groups` overloads are only reachable from tests, and the 3-arg `RecordIndexLookupMetrics.resolveBundle` (L72) has no caller at all. #19063 notes the next executor-metric users (index and MDT read paths) need histogram primitives first, so the second group is likely a different shape. Could we inline this to one concrete class now (also dropping `TestExecutorMetricsGenericity` and the test-only public helpers `RecordIndexMetricNames.registryName` / `DrainedCounters.isEmpty`), and extract the abstraction when a second group lands?



##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/metrics/TestDistributedRegistry.java:
##########
@@ -184,6 +187,131 @@ public void testAddMetricsParallel() {
     Assertions.assertEquals(finalExpectedSum, metricCounts.get(METRIC_1));
   }
 
+  @Test
+  public void testSetThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testSetOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: set() invoked on an executor must fail - it is non-commutative under accumulator merges.
+    // The UnsupportedOperationException thrown on the executor surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.set() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.set(METRIC_1, value);
+          return null;
+        }, 1));
+  }
+
+  @Test
+  public void testReleaseThrowsOnExecutor() {
+    // Given: a registry registered to the spark context
+    String registryName = REGISTRY_NAME + "_testReleaseOnExecutor";
+    Registry registry = engineContext.getMetricRegistry("", registryName);
+
+    List<Integer> data = new ArrayList<>();
+    data.add(1);
+
+    // When/Then: release() invoked on an executor must fail - clamping and eviction are order-dependent
+    // under accumulator merges. The UnsupportedOperationException surfaces wrapped in a SparkException.
+    assertFailsOnExecutorWith("DistributedRegistry.release() must not be called from a Spark executor", () ->
+        engineContext.map(data, value -> {
+          registry.release(Collections.singletonMap(METRIC_1, (long) value));
+          return null;
+        }, 1));
+  }
+
+  /**
+   * Asserts the job failed because the executor-side guard fired, not for some unrelated reason such as a serialization error.
+   */
+  private static void assertFailsOnExecutorWith(String expectedMessage, Executable executable) {
+    SparkException thrown = Assertions.assertThrows(SparkException.class, executable);
+    StringBuilder chain = new StringBuilder();
+    for (Throwable t = thrown; t != null; t = t.getCause()) {
+      chain.append(t).append('\n');
+      if (t.getCause() == t) {
+        break;
+      }
+    }
+    Assertions.assertTrue(chain.toString().contains(expectedMessage),
+        "expected the executor-side guard to fail the job, got: " + chain);
+  }
+
+  @Test
+  public void testSetOnDriverSucceeds() {
+    // set() on the driver (no TaskContext) remains supported.
+    DistributedRegistry registry = new DistributedRegistry(REGISTRY_NAME + "_testSetOnDriver");
+    registry.set(METRIC_1, 42);
+    Assertions.assertEquals(42, registry.getAllCounts().get(METRIC_1));
+  }
+
+  @Test
+  public void testGetMetricRegistryReplacesNonDistributedRegistry() {

Review Comment:
   This test never enters the branch it names: `getMetricRegistry` removes the seeded `LocalRegistry` at `HoodieSparkEngineContext:308` before `getRegistryOfClass` runs, so `computeIfAbsent` creates a fresh `DistributedRegistry` and the `!(instanceof)` fallback at L310 stays uncovered. Could the test seed via `Registry.getRegistryOfClass` from a second thread released after the remove (or the branch be dropped if the instance-scoped registry lands)?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = context.getMetricRegistry(config.getTableName(),
+          metricRegistry.scopedName(config.getBasePath()));
+      // Only the accumulator-backed registry aggregates back to the driver, so anything else is left out
+      // rather than bound: a bound LocalRegistry would collect on the executor and be dropped on the floor,
+      // whereas leaving it out makes the lookup resolve to a no-op that reports nothing.
+      if (registry instanceof DistributedRegistry) {
+        bundle.put(metricRegistry.registryName(), registry);
+      }
+    }
+    return bundle.isEmpty() ? Collections.emptyMap() : bundle;
+  }
+
+  /**
+   * Records one shard's lookup outcome. Counts records rather than distinct keys, so
+   * {@code hits + misses == records_looked_up} holds when a batch repeats a key. Membership is tested
+   * against the found set, bounded by the hit count, not the asked-about set, bounded by shard size.
+   *
+   * @param keysLookedUp every record key routed to this shard
+   * @param foundKeys    the subset present in the index
+   * @param elapsedMs    wall-clock spent reading this shard
+   */
+  public static void recordShardLookup(String caller, Collection<String> keysLookedUp,
+                                       Collection<String> foundKeys, long elapsedMs) {
+    if (keysLookedUp.isEmpty()) {
+      return;
+    }
+    Registry registry = Registry.getRegistry(RecordIndexMetricNames.REGISTRY_NAME);

Review Comment:
   Both callers of `recordShardLookup` already hold `metricsBundle` as a field (`SparkMetadataTableGlobalRecordLevelIndex:214`, `PartitionedRecordIndexFileGroupLookupFunction:92`), so resolving by name here can only return `metricsBundle.get(REGISTRY_NAME)`; nothing shipped emits without the bundle in hand. Passing that `Registry` in directly (null -> return) would let `ExecutorMetricsContext`, `NoOpRegistry`, `TestRegistryExecutorLookup`, the `HoodieSparkTable` binding and the `Registry.getRegistry` contract change go (about 330 lines, both hudi-io additions) with identical behaviour. Could we take that shape and defer name resolution until an emitter that lacks the bundle exists?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/PartitionedRecordIndexFileGroupLookupFunction.java:
##########
@@ -44,36 +47,57 @@ public class PartitionedRecordIndexFileGroupLookupFunction
     implements PairFlatMapFunction<Iterator<Pair<String, String>>, String, HoodieRecordGlobalLocation> {
 
   private final HoodieTableMetadata metadataTable;
+  // Empty when no counters should be collected; see RecordIndexLookupMetrics#resolveBundle.
+  private final Map<String, Registry> metricsBundle;
+  private final String caller;
 
+  /** Uninstrumented, for the query-side read path. */
   public PartitionedRecordIndexFileGroupLookupFunction(HoodieTableMetadata metadataTable) {
+    this(metadataTable, Collections.emptyMap(), null);
+  }
+
+  public PartitionedRecordIndexFileGroupLookupFunction(HoodieTableMetadata metadataTable,
+                                                       Map<String, Registry> metricsBundle, String caller) {
     this.metadataTable = metadataTable;
+    this.metricsBundle = metricsBundle;
+    this.caller = caller;
   }
 
   @Override
   public Iterator<Tuple2<String, HoodieRecordGlobalLocation>> call(Iterator<Pair<String, String>> partitionPathRecordKeyIterator) {
-    String partitionName = null;
-    List<String> keysToLookup = new ArrayList<>();
-    while (partitionPathRecordKeyIterator.hasNext()) {
-      Pair<String, String> partitionPathRecordKey = partitionPathRecordKeyIterator.next();
-      keysToLookup.add(partitionPathRecordKey.getRight());
-      if (partitionName == null) {
-        partitionName = partitionPathRecordKey.getLeft();
+    // Bound for the whole task so a metric raised deeper in the lookup resolves here too.

Review Comment:
   nit, feel free to ignore: the binding is released in the `finally` before Spark consumes the returned iterator, so it covers `call()` (the shard read), not the whole task.
   ```suggestion
       // Bound for the duration of the shard read so a metric raised deeper in the lookup resolves here too.
   ```



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestRliLookupMetricsReporting.scala:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.config.metrics.HoodieMetricsConfig
+import org.apache.hudi.metrics.{ExecutorMetricRegistry, RecordIndexMetricNames}
+
+import org.apache.spark.sql.SaveMode
+import org.junit.jupiter.api.{Tag, Test}
+import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue}
+
+import java.io.{ByteArrayOutputStream, PrintStream}
+
+/** The counters must reach a live metrics reporter, not only commit metadata. */
+@Tag("functional")
+class TestRliLookupMetricsReporting extends RliLookupMetricsTestBase {
+
+  private def metricsOpts: Map[String, String] = rliOpts ++ Map(
+    HoodieMetricsConfig.TURN_METRICS_ON.key -> "true",
+    HoodieMetricsConfig.METRICS_REPORTER_TYPE_VALUE.key -> "CONSOLE")
+
+  /** Captures stdout for the duration of the write. */
+  private def captureStdout(body: => Unit): String = {
+    val buffer = new ByteArrayOutputStream()
+    val original = System.out
+    try {
+      System.setOut(new PrintStream(buffer, true, "UTF-8"))
+      body
+    } finally {
+      System.setOut(original)
+    }
+    buffer.toString("UTF-8")
+  }
+
+  @Test
+  def testCountersReachBothCommitMetadataAndTheReporter(): Unit = {
+    val numUpdates = 12
+
+    doWriteAndValidateDataAndRecordIndex(metricsOpts, INSERT_OPERATION_OPT_VAL, SaveMode.Overwrite,
+      validate = false, numInserts = 50)
+    clearRliRegistry()
+
+    val stdout = captureStdout {
+      doWriteAndValidateDataAndRecordIndex(metricsOpts, UPSERT_OPERATION_OPT_VAL, SaveMode.Append,
+        validate = false, numUpdates = numUpdates)
+    }
+
+    // Sink 1 -- the timeline.
+    val counters = rliCountersFromLatestCommit()
+    report(s"Reporter test ($indexLabel) -- commit metadata", counters)
+    assertTrue(counters.nonEmpty, "the commit must still carry the counters when a reporter is configured")
+    assertEquals(numUpdates.toString, counters(tagKey(RecordIndexMetricNames.KEY_HIT_COUNT)))
+    assertEquals((numUpdates + 1).toLong, assertSumInvariant(counters, RecordIndexMetricNames.CALLER_TAG_LOCATION))
+
+    // Sink 2 -- the reporter. Gauge names are <prefix>.rli.lookup.<caller>.<metric>.
+    val gaugePrefix = s"${ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.metricAction}.${ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.metricQualifier}"
+    val reported = stdout.linesIterator.filter(_.contains(gaugePrefix)).toSeq
+
+    println(s"\n===== Reporter test ($indexLabel) -- ConsoleMetricsReporter output =====")
+    if (reported.isEmpty) println("  (no rli.lookup gauges printed)") else reported.foreach(l => println(s"  ${l.trim}"))
+    println("=======================================================================\n")
+
+    assertTrue(reported.nonEmpty,
+      s"ConsoleMetricsReporter must publish the '$gaugePrefix' gauges; the drain feeds the reporter and " +
+        "commit metadata from a single read, so finding them in the commit but not here means the " +
+        "reporter sink regressed")
+    Seq(RecordIndexMetricNames.KEY_HIT_COUNT, RecordIndexMetricNames.KEY_MISS_COUNT,
+      RecordIndexMetricNames.KEY_COUNT, RecordIndexMetricNames.SHARDS_READ).foreach { metric =>
+      val name = s"$gaugePrefix.${RecordIndexMetricNames.key(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric)}"
+      assertTrue(stdout.contains(name), s"reporter output must contain the gauge '$name'")

Review Comment:
   This checks that the gauge name was printed, never a value, so a regression that publishes zeros or a stale map to `Metrics.registerGauges` passes. The console output has `value = N` on the next line and the expected value is derivable (`numUpdates + 1`). Could the test assert the value too (or, per the consolidation note, move to a `TestHoodieMetrics` unit test that reads the gauge directly)?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la
+ */
+abstract class RliLookupMetricsTestBase extends RecordLevelIndexTestBase {
+
+  /** Overridden by the partitioned subclasses; both variants are separate closures on separate paths. */
+  protected def isPartitionedRli: Boolean = false
+
+  /**
+   * Table type under test. Tagging is an index-level concern and does not branch on table type, so MOR
+   * is expected to behave identically -- the MOR subclasses exist to prove that rather than assume it.
+   */
+  protected def tableTypeOpt: String = DataSourceWriteOptions.COW_TABLE_TYPE_OPT_VAL
+
+  protected def indexLabel: String = {
+    val idx = if (isPartitionedRli) "partitioned RLI" else "global RLI"
+    val tt = if (tableTypeOpt == DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL) "MOR" else "COW"
+    s"$idx, $tt"
+  }
+
+  /**
+   * `commonOpts` turns the global record index on, so the metadata-partition flags and the index type
+   * have to be flipped together to select the partitioned variant.
+   */
+  protected def rliOpts: Map[String, String] = {
+    val withTableType = Map(DataSourceWriteOptions.TABLE_TYPE.key -> tableTypeOpt)
+    if (isPartitionedRli) {
+      commonOpts ++ withTableType ++ Map(
+        HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false",
+        HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true",
+        HoodieIndexConfig.INDEX_TYPE.key -> "RECORD_LEVEL_INDEX")
+    } else {
+      commonOpts ++ withTableType ++ Map(
+        HoodieMetadataConfig.GLOBAL_RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "true",
+        HoodieMetadataConfig.RECORD_LEVEL_INDEX_ENABLE_PROP.key -> "false",
+        HoodieIndexConfig.INDEX_TYPE.key -> "GLOBAL_RECORD_LEVEL_INDEX")
+    }
+  }
+
+  protected def counterKey(caller: String, metric: String): String =
+    RecordIndexMetricNames.COMMIT_METADATA_PREFIX + RecordIndexMetricNames.key(caller, metric)
+
+  protected def tagKey(metric: String): String =
+    counterKey(RecordIndexMetricNames.CALLER_TAG_LOCATION, metric)
+
+  /**
+   * A caller that looked something up stamps its full counter set, zeros included, so an absent key means
+   * that caller contributed nothing at all. The default is defensive against exactly that case.
+   */
+  protected def counterOrZero(counters: Map[String, String], caller: String, metric: String): Long =
+    counters.getOrElse(counterKey(caller, metric), "0").toLong
+
+  /** The counters as an operator would read them: off the latest completed commit. */
+  protected def rliCountersFromLatestCommit(): Map[String, String] = {
+    metaClient.reloadActiveTimeline()
+    val lastInstant = metaClient.getActiveTimeline.getCommitsTimeline.filterCompletedInstants().lastInstant().get()
+    metaClient.getActiveTimeline.readCommitMetadata(lastInstant).getExtraMetadata.asScala.toMap
+      .filter { case (k, _) => k.startsWith(RecordIndexMetricNames.COMMIT_METADATA_PREFIX) }
+  }
+
+  /** Leftover counters from a previous write would otherwise be folded into the next commit. */
+  protected def clearRliRegistry(): Unit = {
+    Registry.REGISTRY_MAP.asScala.foreach {
+      case (key, registry) => if (key.contains(RecordIndexMetricNames.REGISTRY_NAME)) registry.clear()
+    }
+  }
+
+  /** Asserts the core invariant and returns the looked-up count. */
+  protected def assertSumInvariant(counters: Map[String, String], caller: String): Long = {
+    val lookedUp = counterOrZero(counters, caller, RecordIndexMetricNames.KEY_COUNT)
+    val hits = counterOrZero(counters, caller, RecordIndexMetricNames.KEY_HIT_COUNT)
+    val misses = counterOrZero(counters, caller, RecordIndexMetricNames.KEY_MISS_COUNT)
+    org.junit.jupiter.api.Assertions.assertEquals(lookedUp, hits + misses,
+      s"hits + misses must account for every key looked up by '$caller'")
+    // A caller that looked something up must also report the time it took, or the timing metric is
+    // silently absent on paths nobody checked. Zero is allowed: a shard read can round below a millisecond.
+    if (lookedUp > 0) {
+      org.junit.jupiter.api.Assertions.assertTrue(
+        counters.contains(counterKey(caller, RecordIndexMetricNames.LOOKUP_TIME)),
+        s"'$caller' looked up $lookedUp keys but reported no ${RecordIndexMetricNames.LOOKUP_TIME}; " +
+          s"counters were ${counters.keys.toSeq.sorted.mkString(", ")}")
+      org.junit.jupiter.api.Assertions.assertTrue(
+        counterOrZero(counters, caller, RecordIndexMetricNames.LOOKUP_TIME) >= 0L,

Review Comment:
   nit, feel free to ignore: `>= 0L` on an unsigned counter cannot fail (the `contains` above is the real check), and the `report(...)` helpers at L124-134 are `println`s that assert nothing, run 30+ times into CI logs (13 `println`s exist across the other 72 functional Scala files combined). Could both go, relying on the assertion messages that already interpolate `got $counters`?



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {

Review Comment:
   nit, feel free to ignore: this branch is unreachable. `LocalRegistry.getAllCounts` boxes a primitive `long`, and `DistributedRegistry`'s map is a `ConcurrentHashMap`, which forbids null values. Could it go?



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java:
##########
@@ -142,9 +146,25 @@ protected Option<HoodieTableMetadataWriter> getMetadataWriter(
     return Option.empty();
   }
 
+  /**
+   * Carries the task thread's state onto the pool threads Hudi hands record writing to.
+   *
+   * <p>The metrics binding travels alongside the {@link TaskContext} for the same reason the
+   * TaskContext does: both are thread-local, and {@code BoundedInMemoryExecutor} and
+   * {@code DisruptorExecutor} run on threads the task did not create. Without this, a metric raised
+   * while writing records resolves against an unbound thread and is discarded.
+   *
+   * <p>Nothing unbinds afterwards. These pools are created and shut down per operation, so the thread
+   * does not outlive the binding; a pool that were shared across tables would need an unbind hook on
+   * the other side, which no current executor provides.
+   */
   @Override
   public Runnable getPreExecuteRunnable() {
     final TaskContext taskContext = TaskContext.get();
-    return () -> TaskContext$.MODULE$.setTaskContext(taskContext);
+    final Map<String, Registry> metricsBinding = ExecutorMetricsContext.capture();
+    return () -> {
+      TaskContext$.MODULE$.setTaskContext(taskContext);
+      ExecutorMetricsContext.bind(metricsBinding);

Review Comment:
   Confirmed on both counts. Every `getPreExecuteRunnable` call site is write-stage (`HoodieMergeHelper:154`, `SparkLazyInsertIterable:77`, the bootstrap handlers, `SingleSparkJobConsistentHashingExecutionStrategy:132/217`), a different stage from the lookup, and both lookup functions `unbind` in a `finally`, so `capture()` returns `emptyMap` and the only effect of `bind(emptyMap)` is `isBound() == true`, which turns every `Registry.getRegistry` miss on those pool threads into `NoOpRegistry`. Inert today (no production `getRegistry` runs on executors) and untested. Could it be dropped until something actually emits from a writer-pool thread?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858501546


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {
+          continue;
+        }
+        counts.put(counter.getKey(), counter.getValue());
+        recordedSomething |= counter.getValue() != 0L;
+      }
+      if (!recordedSomething) {

Review Comment:
   Ideally this should resolve as discussed with @yihua and will be keeping this intial phase for metrics reporter only.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3858485380


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry implements ExecutorMetricGroup {
+
+  RECORD_INDEX_LOOKUP(
+      "HoodieRecordIndexLookup",
+      "hoodie.rli.lookup.",

Review Comment:
   I think i will follow the convention for name thats we use in metrics reporter related classes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3859015366


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = context.getMetricRegistry(config.getTableName(),

Review Comment:
   Fixed. `TBL_NAME` has no default and `Builder.validate()` only requires `BASE_PATH`, so a config built without `forTable()` reached `getMetricRegistry` with a null name and NPE'd on `tableName.isEmpty()`. Took the suggestion, skipping collection when the name is null or empty.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/DistributedRegistry.java:
##########
@@ -68,9 +79,36 @@ public void add(String name, long value) {
 
   @Override
   public void set(String name, long value) {
+    // Last-writer-wins is neither commutative nor associative, and the driver merges executor copies in
+    // an unspecified order. Driver only; executors use increment()/add().
+    if (TaskContext.get() != null) {

Review Comment:
   Fixed, and the point generalises. Both guards now `LOG.warn` and return instead of throwing, and the whole drain at the commit boundary is wrapped in a catch-all that logs and moves on. It runs after the commit has landed, so no reporting problem should be able to take a completed write down with it -- same reason `Metrics.registerGauge` already swallows.
   
   The two tests that asserted the throw now assert the behaviour that actually matters: the job succeeds, and `set()` records nothing while `release()` leaves the counters alone. That also drops the exception-message matching you flagged separately.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {

Review Comment:
   Fixed both. The remover now also drops the `Registry.makeKey("", prefixedName)` entry, and is renamed `removeMetricRegistry` with `@VisibleForTesting`.
   
   For what it is worth, this reproduced exactly as described: instrumenting the drain and dumping `REGISTRY_MAP` shows both `hoodie_test::HoodieRecordIndexLookup.<digest>` and `::hoodie_test.HoodieRecordIndexLookup.<digest>` present simultaneously.



##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = context.getMetricRegistry(config.getTableName(),
+          metricRegistry.scopedName(config.getBasePath()));
+      // Only the accumulator-backed registry aggregates back to the driver, so anything else is left out
+      // rather than bound: a bound LocalRegistry would collect on the executor and be dropped on the floor,
+      // whereas leaving it out makes the lookup resolve to a no-op that reports nothing.
+      if (registry instanceof DistributedRegistry) {
+        bundle.put(metricRegistry.registryName(), registry);
+      }
+    }
+    return bundle.isEmpty() ? Collections.emptyMap() : bundle;
+  }
+
+  /**
+   * Records one shard's lookup outcome. Counts records rather than distinct keys, so
+   * {@code hits + misses == records_looked_up} holds when a batch repeats a key. Membership is tested
+   * against the found set, bounded by the hit count, not the asked-about set, bounded by shard size.
+   *
+   * @param keysLookedUp every record key routed to this shard
+   * @param foundKeys    the subset present in the index
+   * @param elapsedMs    wall-clock spent reading this shard
+   */
+  public static void recordShardLookup(String caller, Collection<String> keysLookedUp,

Review Comment:
   Fixed. `recordShardLookup` now returns as soon as the resolved registry is a `NoOpRegistry`, before the O(keys) hit scan. That covers the query read path and the disabled path. The `null.<metric>` keys are gone too, since caller tagging has been removed from the PR entirely.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metrics/ExecutorMetrics.java:
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Commit-boundary drain for executor-collected metrics, generic over {@link ExecutorMetricRegistry}. On
+ * the shared commit path, so it covers Spark DataSource, Spark SQL and DeltaStreamer alike.
+ */
+public class ExecutorMetrics {
+
+  private ExecutorMetrics() {
+  }
+
+  /**
+   * Snapshots into commit metadata without consuming. Split from {@link #publishAndRelease} so a commit
+   * that never lands neither loses its counters nor publishes gauges for rolled-back work. An all-zero
+   * registry is skipped to keep residue off the timeline; zeros are otherwise kept, since an explicit
+   * {@code misses=0} is meaningful.
+   */
+  public static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                           HoodieWriteConfig config) {
+    return snapshotIntoCommitMetadata(commitMetadata, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the collection machinery against a group it does not ship with. */
+  static DrainedCounters snapshotIntoCommitMetadata(Map<String, String> commitMetadata,
+                                                    HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    List<Drained> drained = new ArrayList<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = Registry.REGISTRY_MAP.get(
+          Registry.makeKey(config.getTableName(), metricRegistry.scopedName(config.getBasePath())));
+      if (registry == null) {
+        continue;
+      }
+      Map<String, Long> counts = new HashMap<>();
+      boolean recordedSomething = false;
+      for (Map.Entry<String, Long> counter : registry.getAllCounts(false).entrySet()) {
+        if (counter.getValue() == null) {

Review Comment:
   Fixed -- you are right that it cannot be null (`LocalRegistry` boxes a primitive `long`, and `ConcurrentHashMap` forbids null values). Simplified to `allMatch(value -> value == 0L)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): Report record index lookup key counts and hit rate per commit [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3840643857


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {
+    DISTRIBUTED_REGISTRY_MAP.remove(tableName.isEmpty() ? registryName : tableName + "." + registryName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {
     final String prefixedName = tableName.isEmpty() ? registryName : tableName + "." + registryName;
-    return DISTRIBUTED_REGISTRY_MAP.computeIfAbsent(prefixedName, key -> {
+    // Both maps are process-wide statics that outlive any SparkContext, so the staleness check and the

Review Comment:
   same here about comments being too verbose



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3862985473


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricGroup;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  /** Set by the read client around its own tagging call, so dedupe traffic is attributable separately. */
+  private static final ThreadLocal<String> CALLER =
+      ThreadLocal.withInitial(() -> RecordIndexMetricNames.CALLER_TAG_LOCATION);
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  public static String currentCaller() {
+    return CALLER.get();
+  }
+
+  /** Restore rather than clear, so a nested tagging call does not reset the label. */
+  public static String setCaller(String caller) {
+    String previous = CALLER.get();
+    CALLER.set(caller);
+    return previous;
+  }
+
+  public static void restoreCaller(String previous) {
+    CALLER.set(previous);
+  }
+
+  /**
+   * The registries a lookup task collects into, keyed by bare name. Includes every entry on
+   * {@link ExecutorMetricRegistry}. Delivery is by closure capture, which is deterministic; resolution is
+   * by name, which lets code below the write API take part without a signature change.
+   */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config) {
+    return resolveBundle(context, config, Arrays.asList(ExecutorMetricRegistry.values()));
+  }
+
+  /** Visible for testing the bundle against a group the enum does not ship with. */
+  public static Map<String, Registry> resolveBundle(HoodieEngineContext context, HoodieWriteConfig config,
+                                                    Collection<? extends ExecutorMetricGroup> groups) {
+    Map<String, Registry> bundle = new HashMap<>();
+    for (ExecutorMetricGroup metricRegistry : groups) {
+      if (!metricRegistry.isEnabled(config)) {
+        continue;
+      }
+      Registry registry = context.getMetricRegistry(config.getTableName(),
+          metricRegistry.scopedName(config.getBasePath()));
+      // Only the accumulator-backed registry aggregates back to the driver, so anything else is left out
+      // rather than bound: a bound LocalRegistry would collect on the executor and be dropped on the floor,
+      // whereas leaving it out makes the lookup resolve to a no-op that reports nothing.
+      if (registry instanceof DistributedRegistry) {
+        bundle.put(metricRegistry.registryName(), registry);
+      }
+    }
+    return bundle.isEmpty() ? Collections.emptyMap() : bundle;
+  }
+
+  /**
+   * Records one shard's lookup outcome. Counts records rather than distinct keys, so
+   * {@code hits + misses == records_looked_up} holds when a batch repeats a key. Membership is tested
+   * against the found set, bounded by the hit count, not the asked-about set, bounded by shard size.
+   *
+   * @param keysLookedUp every record key routed to this shard
+   * @param foundKeys    the subset present in the index
+   * @param elapsedMs    wall-clock spent reading this shard
+   */
+  public static void recordShardLookup(String caller, Collection<String> keysLookedUp,
+                                       Collection<String> foundKeys, long elapsedMs) {
+    if (keysLookedUp.isEmpty()) {
+      return;
+    }
+    Registry registry = Registry.getRegistry(RecordIndexMetricNames.REGISTRY_NAME);

Review Comment:
   Sure, however, i do not see a followup issue for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3863379970


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RliLookupMetricsTestBase.scala:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.functional
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.common.config.HoodieMetadataConfig
+import org.apache.hudi.common.metrics.Registry
+import org.apache.hudi.config.HoodieIndexConfig
+import org.apache.hudi.metrics.RecordIndexMetricNames
+
+import scala.collection.JavaConverters._
+
+/**
+ * Shared plumbing for the record level index lookup metric tests: index selection, and reading the counters back the way an operator would -- off the la
+ */
+abstract class RliLookupMetricsTestBase extends RecordLevelIndexTestBase {

Review Comment:
   Moved to #19759 (section D) to be cut together with the lifetime change, as you suggested. Resolving.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3862990025


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,43 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistryForTesting(String tableName, String registryName) {
+    DISTRIBUTED_REGISTRY_MAP.remove(tableName.isEmpty() ? registryName : tableName + "." + registryName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {

Review Comment:
   Agreed on scoping it out. Since [16](https://github.com/apache/hudi/pull/19575#discussion_r3853269022), [24](https://github.com/apache/hudi/pull/19575#discussion_r3853269071), [39](https://github.com/apache/hudi/pull/19575#discussion_r3853269157), [41](https://github.com/apache/hudi/pull/19575#discussion_r3853269168) and the `Metrics.shutdown` scrape on the `ExecutorMetricRegistry:78` thread all defer to this same change and none of them has a tracking artifact yet, could you add one checklist line to #19063 listing them, so they are not lost when this merges?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5433521825

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 0a9d4b412caeb2ef3dc19c1b16287265bd106fe7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568) 
   * eadd7bbf6072d5c29c93f1650ae25f2b5f4de998 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3868350282


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry {

Review Comment:
   This class is in spark client already



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3868350282


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metrics/ExecutorMetricRegistry.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metrics;
+
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.storage.StoragePath;
+
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.util.function.Predicate;
+
+/**
+ * Every class of executor-collected metric, and the only thing a new one is added to. The driver must
+ * declare it up front because an {@code AccumulatorV2} must be registered with the {@code SparkContext}
+ * before a task can contribute; the bundle sent to executors and the commit drain both iterate this.
+ */
+public enum ExecutorMetricRegistry {

Review Comment:
   This class is in spark client already, i have moved the other classes however



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): make record index lookups observable via executor metrics [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5348545039

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 369fbd591652f70be61c8c5fa19526955d41ba38 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260) 
   * f949227bee97d7b41d87d8c18ff73f6d15d7de17 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): make record index lookups observable via executor metrics [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5348593637

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 369fbd591652f70be61c8c5fa19526955d41ba38 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260) 
   * f949227bee97d7b41d87d8c18ff73f6d15d7de17 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "rahil-c (via GitHub)" <gi...@apache.org>.
rahil-c commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3874972540


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/RecordIndexLookupMetrics.java:
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.index;
+
+import org.apache.hudi.common.engine.HoodieEngineContext;
+import org.apache.hudi.common.metrics.Registry;
+import org.apache.hudi.config.HoodieWriteConfig;
+import org.apache.hudi.metrics.DistributedRegistry;
+import org.apache.hudi.metrics.ExecutorMetricRegistry;
+import org.apache.hudi.metrics.RecordIndexMetricNames;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Set;
+
+/** Executor-side emission for the record index lookup counters. */
+public class RecordIndexLookupMetrics {
+
+  private RecordIndexLookupMetrics() {
+  }
+
+  /**
+   * The registry a lookup task collects into, or null when nothing should be collected. Captured in the
+   * lookup closure and passed to {@link #recordShardLookup}, so delivery is by closure capture rather
+   * than by name.
+   *
+   * <p>Requires the reporter to be on as well: with {@code hoodie.metrics.on} off there is nowhere to
+   * publish, and collecting would register an accumulator and scan every shard for nothing.
+   */
+  public static Registry resolveRegistry(HoodieEngineContext context, HoodieWriteConfig config) {
+    if (!config.isMetricsOn() || !ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.isEnabled(config)) {
+      return null;
+    }
+    // TBL_NAME has no default and Builder.validate() only requires BASE_PATH, so a config built without
+    // forTable() reaches here with a null name. getMetricRegistry dereferences it immediately.
+    String tableName = config.getTableName();
+    if (tableName == null || tableName.isEmpty()) {
+      return null;
+    }
+    Registry registry = context.getMetricRegistry(tableName,
+        ExecutorMetricRegistry.RECORD_INDEX_LOOKUP.scopedName(config.getBasePath()));
+    // Only the accumulator-backed registry aggregates back to the driver. A LocalRegistry here would
+    // collect on the executor and be dropped on the floor, so report nothing instead.
+    if (!(registry instanceof DistributedRegistry)) {
+      return null;
+    }
+    // Runs on the driver before the closure ships, so anything held now predates this write. Publishing
+    // releases what it reports, which leaves a non-empty registry only after an attempt that never
+    // committed. Paths that tear metrics down between writes lose it anyway; Spark SQL DML and StreamSync
+    // do not, and this commit must not report work it did not do.
+    registry.clear();

Review Comment:
   One scoping point that I think bounds it. Every piece of state involved is per-JVM:
   
   - `Registry.REGISTRY_MAP` is an interface field, so implicitly `public static final`
   - `HoodieSparkEngineContext.DISTRIBUTED_REGISTRY_MAP` is `private static final`
   - the counters are an instance field on the `DistributedRegistry`, and merging goes through the task-result envelope of the `SparkContext` it was registered with
   
   Nothing is persisted outside the process either. The commit-metadata sink was removed earlier in this PR, so the only outbound path is `registerGauges`.
   
   So two write clients in one JVM do share a bucket and can clear and publish each other's counts, exactly as you describe. Two separate applications writing the same table cannot: separate drivers, separate static maps, separate accumulators, and an executor in one application has no route to the other's accumulator.
   
   On disabling for multiwriter: the gate would have to key off `WriteConcurrencyMode.supportsMultiWriter()`, which cannot distinguish separate-application multiwriter from in-process. It would switch the feature off for deployments that are not affected, including the one that asked for it, while the case it guards is the narrower in-process one.
   
   On scoping: instant-based scoping is not reachable from where the registry is resolved. `lookupRecords(records, context, hoodieTable, fileGroupSize)` carries no instant, and `HoodieTable` takes it as a parameter rather than holding it. A per-writer key is reachable in about eight lines, by stamping a UUID into the write config at client construction and including it in the registry key. The cost is that the key can then only be reconstructed from a live client config, which breaks the test helper that seeds the abandoned-attempt regression case.
   
   Proposal: take this as a follow-up scoped on whether in-process concurrent writers are actually in use, and document the limitation on the config in the meantime. It sits naturally with the registry-lifetime work already tracked in #19759, which removes the process-wide map and makes per-writer scoping fall out of it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19575:
URL: https://github.com/apache/hudi/pull/19575#discussion_r3877756560


##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/common/HoodieSparkEngineContext.java:
##########
@@ -278,11 +279,47 @@ public String getApplicationId() {
     return javaSparkContext.sc().applicationId();
   }
 
+  /**
+   * Drops a registry from both process-wide maps. Only for tests that create their own SparkContexts:
+   * without it they leave accumulators bound to stopped contexts behind for whatever runs next in the
+   * same JVM.
+   */
+  @VisibleForTesting
+  public static void removeMetricRegistry(String tableName, String registryName) {
+    String prefixedName = tableName.isEmpty() ? registryName : tableName + "." + registryName;
+    DISTRIBUTED_REGISTRY_MAP.remove(prefixedName);
+    Registry.REGISTRY_MAP.remove(Registry.makeKey(tableName, registryName));
+    // setRegistries also indexes under the empty table name, so evicting only the table-scoped key leaves
+    // the accumulator reachable under ::<table>.<registry>.
+    Registry.REGISTRY_MAP.remove(Registry.makeKey("", prefixedName));
+  }
+
   @Override
   public Registry getMetricRegistry(String tableName, String registryName) {
     final String prefixedName = tableName.isEmpty() ? registryName : tableName + "." + registryName;
-    return DISTRIBUTED_REGISTRY_MAP.computeIfAbsent(prefixedName, key -> {
+    // Both maps are process-wide statics that outlive any SparkContext, so the staleness check and the
+    // recreation have to be atomic: otherwise one caller can evict the registry another caller just
+    // created, leaving two live accumulators for one metric name while reporting only ever reads the
+    // one still in the map.
+    return DISTRIBUTED_REGISTRY_MAP.compute(prefixedName, (key, cached) -> {
+      if (cached instanceof DistributedRegistry && ((DistributedRegistry) cached).isRegisteredWith(javaSparkContext)) {
+        return cached;
+      }
+      // Nothing usable cached, or the cached accumulator is bound to a SparkContext that is no longer
+      // live (a restart in the same JVM: shells, notebooks, Spark Connect). Drop the shared-map entry
+      // first, since getRegistryOfClass() would otherwise hand back that same stale instance.
+      final String sharedKey = Registry.makeKey(tableName, registryName);
+      Registry.REGISTRY_MAP.remove(sharedKey);

Review Comment:
   Can we hide these registry details from spark context:
   
   1. make `getRegistryOfClass` thread safe;
   2. add a param or new API to support overide instead of existing `computeIfAbsent`;



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5448774282

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     }, {
       "hash" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644",
       "triggerID" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bc8688a911937a4a0fef69304f182697141245d9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bc8688a911937a4a0fef69304f182697141245d9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * 695f74f6364193e7869f83b7eab88e9b637b555c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644) 
   * bc8688a911937a4a0fef69304f182697141245d9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters to the metrics reporter [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5449410641

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16438",
       "triggerID" : "c3df340e3609cb1ad7f5f8abce5d207625887fa7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16442",
       "triggerID" : "457b1ac5c6ca7b2e46816e9c22d027bb498dbe41",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16468",
       "triggerID" : "8a71f71978edc6f83d37711a582e516b743aa7e7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16555",
       "triggerID" : "e62a131d0a4821a474fc78f037df1c82fcc1c559",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e283481080eb2ed6205c850c08e45a0e2844d412",
       "triggerType" : "PUSH"
     }, {
       "hash" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16556",
       "triggerID" : "221b2b5bfec5eb6548462bd91fe85b3502f52637",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ee19c8b5b4132001edb248e35416350b49007400",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ee19c8b5b4132001edb248e35416350b49007400",
       "triggerType" : "PUSH"
     }, {
       "hash" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16560",
       "triggerID" : "308c956e2a5895f99806e8d2d242f318cf670c42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16563",
       "triggerID" : "bf51bb77e90f1cfa0fed51257b7dd1cc2052a830",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16568",
       "triggerID" : "0a9d4b412caeb2ef3dc19c1b16287265bd106fe7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16606",
       "triggerID" : "eadd7bbf6072d5c29c93f1650ae25f2b5f4de998",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16610",
       "triggerID" : "6412a51abe4a0cbaaf71cf4c9f0e59a10b04f1b7",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16612",
       "triggerID" : "0ae8343d75734b1d9d89e04f1065337fc97faf9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16638",
       "triggerID" : "47a7be9e0ce5d6aec98bb7cd582222a5d754462f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16639",
       "triggerID" : "fa7936408666ceed9010ee10a19b38fa04bec224",
       "triggerType" : "PUSH"
     }, {
       "hash" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16644",
       "triggerID" : "695f74f6364193e7869f83b7eab88e9b637b555c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bc8688a911937a4a0fef69304f182697141245d9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16651",
       "triggerID" : "bc8688a911937a4a0fef69304f182697141245d9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e283481080eb2ed6205c850c08e45a0e2844d412 UNKNOWN
   * ee19c8b5b4132001edb248e35416350b49007400 UNKNOWN
   * bc8688a911937a4a0fef69304f182697141245d9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16651) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5274450267

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f2ace685b8450e4466990371e11deba541aa992 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234) 
   * bb09a2d96ee77c288bc295a9195cfca97a81e3bd UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5363463003

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 70968d47931e1e491260eccd0eadd85e12a83202 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426) 
   * 3d9cb9c3c5e0631abdad072dd0a6706ac42d4521 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428) 
   * 30dbc00d04a21cd671b1e02b527f3b340f828008 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup key counts and hit rate per commit [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5364264111

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16391",
       "triggerID" : "f949227bee97d7b41d87d8c18ff73f6d15d7de17",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16426",
       "triggerID" : "70968d47931e1e491260eccd0eadd85e12a83202",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16428",
       "triggerID" : "3d9cb9c3c5e0631abdad072dd0a6706ac42d4521",
       "triggerType" : "PUSH"
     }, {
       "hash" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431",
       "triggerID" : "30dbc00d04a21cd671b1e02b527f3b340f828008",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432",
       "triggerID" : "2c37a62e311d57eb3636dc5f28af1739788c98f9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435",
       "triggerID" : "a061dbc0124d8ebc62defdf41d2d428842a6b16c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 30dbc00d04a21cd671b1e02b527f3b340f828008 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16431) 
   * 2c37a62e311d57eb3636dc5f28af1739788c98f9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16432) 
   * a061dbc0124d8ebc62defdf41d2d428842a6b16c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16435) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(metrics): report record index lookup counters at the commit boundary [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19575:
URL: https://github.com/apache/hudi/pull/19575#issuecomment-5275091758

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16207",
       "triggerID" : "f0fd521723cb8a9ce999369619fe454e5ca47b4c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16211",
       "triggerID" : "c4fa5fad9b2a0ec0866cd13778d7288c22275494",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16227",
       "triggerID" : "5d3fa9e8ab489adcd8b266911cfaee7672ab8846",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f2ace685b8450e4466990371e11deba541aa992",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16234",
       "triggerID" : "4f2ace685b8450e4466990371e11deba541aa992",
       "triggerType" : "PUSH"
     }, {
       "hash" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259",
       "triggerID" : "bb09a2d96ee77c288bc295a9195cfca97a81e3bd",
       "triggerType" : "PUSH"
     }, {
       "hash" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260",
       "triggerID" : "369fbd591652f70be61c8c5fa19526955d41ba38",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * bb09a2d96ee77c288bc295a9195cfca97a81e3bd Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16259) 
   * 369fbd591652f70be61c8c5fa19526955d41ba38 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16260) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org