You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2025/04/30 20:53:09 UTC

[PR] Introduce EventExecutorMetrics instead of set the metrics props unsafely [ozone]

peterxcli opened a new pull request, #8371:
URL: https://github.com/apache/ozone/pull/8371

   ## What changes were proposed in this pull request?
   
   The error
   ```java
   2025-04-24 16:22:54,652 [main] ERROR utils.MetricsUtil (MetricsUtil.java:updateAnnotation(97)) - Update Metrics annotation failed. 
   java.lang.reflect.InaccessibleObjectException: Unable to make private java.lang.Class$AnnotationData java.lang.Class.annotationData() accessible: module java.base does not "opens java.lang" to unnamed module @2c7b84de
   ```
   keeps appearing in test log
   
   And the root cause and solution can be found from https://stackoverflow.com/a/41265267
   >The exception is caused by the [Java Platform Module System](http://openjdk.java.net/projects/jigsaw/spec/sotms) that was introduced in Java 9, particularly its implementation of strong encapsulation. It only allows [access](http://openjdk.java.net/projects/jigsaw/spec/sotms/#accessibility) under certain conditions, 
   > - the most prominent ones are: the type has to be public
   > - the owning package has to be exported
   
   And I had tried add some argument from the suggestion:
   ```bash
   # --add-opens has the following syntax: {A}/{package}={B}
   java --add-opens java.base/java.lang=ALL-UNNAMED
   ```
   
   But the exception still appear in the test log, I think it might also not a good long term solution. So I created a common metrics class for those two type of event related thread executor. Removing the metrics code from them and the metrics also support customize name for every of its instance.
   
   
   ### Check that exception log is gone
   
   Ran some example test
   ```bash
   mvn test -pl hadoop-ozone/integration-test "-Dtest=TestValidateBCSIDOnRestart#testValidateBCSIDOnDnRestart"
   ```
   
   test output:
   
   [org.apache.hadoop.ozone.client.rpc.TestValidateBCSIDOnRestart-output.txt](https://github.com/user-attachments/files/19984184/org.apache.hadoop.ozone.client.rpc.TestValidateBCSIDOnRestart-output.txt)
   
   
   ### Metrics name correctness
   
   ```bash
   ozone freon stb --scm-host=scm --benchmark=AllocateBlocks
   ```
   ![image](https://github.com/user-attachments/assets/c1010278-cb4b-4d79-983d-bf4e623b2392)
   
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-12901
   
   ## How was this patch tested?
   
   CI:
   https://github.com/peterxcli/ozone/actions/runs/14763747318


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12901. Introduce EventExecutorMetrics instead of setting the metrics props unsafely [ozone]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #8371:
URL: https://github.com/apache/ozone/pull/8371#discussion_r2069884829


##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/events/SingleThreadExecutor.java:
##########
@@ -77,42 +59,43 @@ public SingleThreadExecutor(String name, String threadNamePrefix) {
   @Override
   public void onMessage(EventHandler<P> handler, P message, EventPublisher
       publisher) {
-    queued.incr();
+    metrics.incrementQueued();
     executor.execute(() -> {
-      scheduled.incr();
+      metrics.incrementScheduled();
       try {
         handler.onMessage(message, publisher);
-        done.incr();
+        metrics.incrementDone();
       } catch (Exception ex) {
         LOG.error("Error on execution message {}", message, ex);
-        failed.incr();
+        metrics.incrementFailed();
       }
     });
   }
 
   @Override
   public long failedEvents() {
-    return failed.value();
+    return metrics.getFailed();
   }
 
   @Override
   public long successfulEvents() {
-    return done.value();
+    return metrics.getDone();
   }
 
   @Override
   public long queuedEvents() {
-    return queued.value();
+    return metrics.getQueued();
   }
 
   @Override
   public long scheduledEvents() {
-    return scheduled.value();
+    return metrics.getScheduled();
   }
 
   @Override
   public void close() {

Review Comment:
   The `eventQueue` is also explicitly closed by `StorageContainerManager`:  
   https://github.com/apache/ozone/blob/5cb6dd84d89f962e01500109db89f0da0ff84d29/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java#L1694
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12901. Introduce EventExecutorMetrics instead of setting the metrics props unsafely [ozone]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8371:
URL: https://github.com/apache/ozone/pull/8371#issuecomment-2846419956

   Thanks @jojochuang for reviewing and merging!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12901. Introduce EventExecutorMetrics instead of setting the metrics props unsafely [ozone]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang merged PR #8371:
URL: https://github.com/apache/ozone/pull/8371


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12901. Introduce EventExecutorMetrics instead of set the metrics props unsafely [ozone]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #8371:
URL: https://github.com/apache/ozone/pull/8371#issuecomment-2843262579

   trigger


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12901. Introduce EventExecutorMetrics instead of setting the metrics props unsafely [ozone]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #8371:
URL: https://github.com/apache/ozone/pull/8371#discussion_r2069882219


##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/events/SingleThreadExecutor.java:
##########
@@ -77,42 +59,43 @@ public SingleThreadExecutor(String name, String threadNamePrefix) {
   @Override
   public void onMessage(EventHandler<P> handler, P message, EventPublisher
       publisher) {
-    queued.incr();
+    metrics.incrementQueued();
     executor.execute(() -> {
-      scheduled.incr();
+      metrics.incrementScheduled();
       try {
         handler.onMessage(message, publisher);
-        done.incr();
+        metrics.incrementDone();
       } catch (Exception ex) {
         LOG.error("Error on execution message {}", message, ex);
-        failed.incr();
+        metrics.incrementFailed();
       }
     });
   }
 
   @Override
   public long failedEvents() {
-    return failed.value();
+    return metrics.getFailed();
   }
 
   @Override
   public long successfulEvents() {
-    return done.value();
+    return metrics.getDone();
   }
 
   @Override
   public long queuedEvents() {
-    return queued.value();
+    return metrics.getQueued();
   }
 
   @Override
   public long scheduledEvents() {
-    return scheduled.value();
+    return metrics.getScheduled();
   }
 
   @Override
   public void close() {

Review Comment:
   I guess they are closed at:
   
   https://github.com/apache/ozone/blob/5cb6dd84d89f962e01500109db89f0da0ff84d29/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/events/EventQueue.java#L286-L302
   
   Those executors are init and bind to to the map when calling `executors.putIfAbsent(event, new HashMap<>()); `:
   https://github.com/apache/ozone/blob/5cb6dd84d89f962e01500109db89f0da0ff84d29/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/events/EventQueue.java#L158-L175



##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/events/SingleThreadExecutor.java:
##########
@@ -77,42 +59,43 @@ public SingleThreadExecutor(String name, String threadNamePrefix) {
   @Override
   public void onMessage(EventHandler<P> handler, P message, EventPublisher
       publisher) {
-    queued.incr();
+    metrics.incrementQueued();
     executor.execute(() -> {
-      scheduled.incr();
+      metrics.incrementScheduled();
       try {
         handler.onMessage(message, publisher);
-        done.incr();
+        metrics.incrementDone();
       } catch (Exception ex) {
         LOG.error("Error on execution message {}", message, ex);
-        failed.incr();
+        metrics.incrementFailed();
       }
     });
   }
 
   @Override
   public long failedEvents() {
-    return failed.value();
+    return metrics.getFailed();
   }
 
   @Override
   public long successfulEvents() {
-    return done.value();
+    return metrics.getDone();
   }
 
   @Override
   public long queuedEvents() {
-    return queued.value();
+    return metrics.getQueued();
   }
 
   @Override
   public long scheduledEvents() {
-    return scheduled.value();
+    return metrics.getScheduled();
   }
 
   @Override
   public void close() {

Review Comment:
   I think they are closed at:
   
   https://github.com/apache/ozone/blob/5cb6dd84d89f962e01500109db89f0da0ff84d29/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/events/EventQueue.java#L286-L302
   
   Those executors are init and bind to to the map when calling `executors.putIfAbsent(event, new HashMap<>()); `:
   https://github.com/apache/ozone/blob/5cb6dd84d89f962e01500109db89f0da0ff84d29/hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/events/EventQueue.java#L158-L175



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-12901. Introduce EventExecutorMetrics instead of setting the metrics props unsafely [ozone]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8371:
URL: https://github.com/apache/ozone/pull/8371#discussion_r2069873000


##########
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/server/events/SingleThreadExecutor.java:
##########
@@ -77,42 +59,43 @@ public SingleThreadExecutor(String name, String threadNamePrefix) {
   @Override
   public void onMessage(EventHandler<P> handler, P message, EventPublisher
       publisher) {
-    queued.incr();
+    metrics.incrementQueued();
     executor.execute(() -> {
-      scheduled.incr();
+      metrics.incrementScheduled();
       try {
         handler.onMessage(message, publisher);
-        done.incr();
+        metrics.incrementDone();
       } catch (Exception ex) {
         LOG.error("Error on execution message {}", message, ex);
-        failed.incr();
+        metrics.incrementFailed();
       }
     });
   }
 
   @Override
   public long failedEvents() {
-    return failed.value();
+    return metrics.getFailed();
   }
 
   @Override
   public long successfulEvents() {
-    return done.value();
+    return metrics.getDone();
   }
 
   @Override
   public long queuedEvents() {
-    return queued.value();
+    return metrics.getQueued();
   }
 
   @Override
   public long scheduledEvents() {
-    return scheduled.value();
+    return metrics.getScheduled();
   }
 
   @Override
   public void close() {

Review Comment:
   I'm looking for the usage of these two thread executors and whether or not they are closed explicitly.
   It looks like they are not closed explicitly so this may not be called properly. Probably ok now because they are only initialized once for Recon and SCM.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org